Welcome to Foxite.COM Community Weblog Sign in | Join | Help

Our DevTeach Experience Follow-Up

I have one more point to add in respect of our Vancouver DevTeach experience in June of this year. If you read my blog last week you will know that I had a complaint about the lack of conference materials in general and specifically that:

There was also no conference CD (though I did get an Email telling me that session materials would be downloadable from 6/18)

Well, I just downloaded the session materials for the sessions I wanted and I am seriously angry - for several reasons: 

  • There is no composite download. You have to go to the web site and download each sessions's materials individually.  There are not even files by track - only by session. What a waste of time!
  • Even the session file(s) are not consistent - some have 1 file, some have a dozen. Why couldn't the presenters have zipped their material into 1 file per session?
  • On some sessions, when you try to download the file you get a "Page Not Found" error, other sessions simply have a "No materials available" message
  • When you do get something it is, by and large, useless. The vast majority of "Session Materials" turn out to be just the PowerPoint Slides! The few that have anything more are simply unannotated demo files
  • I spent more than an hour painstakingly downloading over 50 files, one a time, in the hope of finally getting some value - but all I have gotten are a bunch of PowerPoint slides and the odd bit of sample code

There are NO white papers! Not one! Without a white paper a session is pretty much a waste of time.

What I mean is that at the conference you attend 15+ sessions in three days - how can anyone possibly remember all the details? You NEED the white papers so that later, when you come back to review the material, or look up the details of some vaguely remembered speaker's comment, you have some chance of finding it!

In all my years as a conference speaker I have never given a session that did not include a background paper that could be used by attendees to review the session contents and give meaning and context in retrospect. In fact I have never heard of a conference where the submission of a white paper was not part of the requirements for speakers (and I even have known cases where failure to submit a paper meant you didn't speak at all and that you were dropped from the list!)

How can a conference organization be so appallingly dismissive of their attendees?

I really feel we were ripped off from A-Z by this conference and I have already said that we would probably not attend another DevTeach - now I am sure we won't.

In fact, nothing on earth would persuade me to attend another DevTeach and unless you especially like to waste money, I strongly urge you to try another conference where, perhaps, your money and time are appreciated sufficiently by the organizers that they will make a token effort to ensure that you get reasonable value.

The one thing this has experience has taught me is that DevTeach does NOT deliver value by any reasonable measure.

posted by andykr | 2 Comments
Filed Under: ,

Creating Data Driven Pop-Up menus in VFP

One of the little tools that Marcia and I use all the time when working in VFP is a pop-up menu generator that allows us to select a development environment easily and quickly. Yes, we know all about the Task Pane but personally I have always found that a real pain (pardon the pun) to use. All we wanted was a quick and easy way to be able to switch between my various environments.

It seemed to us that the easiest way to do that is to have a pop-up menu that could be invoked from a hot key. Then we realized that there are all sorts of situations in which it would be useful to have a pop-up menu (like right-click options on forms and controls) and so we came up with a simple menu generator to build a pop-up menu on the fly from simple tables. In order to make this generic we set up a relational structure using three tables as shown in Table 1.

These tables are all contained in their own DBC named (imaginateively) "PopMenu". So how do we use the tables?

Well, first we create the records in the names table, currently I have two menus defined, one named "Projects" and another named "EditOptions". The first is the one that I run when I want to select a project environment, and the second is a generic editing options popup. Of course there is no real limit to the number of menus you can define but for the purposes of this article let's stick to these two.

Now we need to define our menu bars for these options and this is really simple. The first thing we need is a horizontal menu divider (grouped options always look better) so that gets defined as the first bar #1 as follows:

cBarText     = "\-"
cBarDesc     = "Divider Line"

Nothing else is needed for this entry. Now we need a bar for each option that we want to include, but for these bars we also need to include the action (and Skip For commands if needed). For my default development environment I use the following:

cBarText     = "\-"
cBarDesc     = "Divider Line"
mBarAction  = " CLOSE ALL
                       CLEAR
                       SET PATH TO (HOME() + ";D:\VFP90\RUN\;LIBS;FORMS;DATA;PROGS;UTILS;BMPS" )
                       SET DEFAULT TO D:\VFP90\RUN\"

And for the "Paste" option on the EditOptions menu we need to include a SKIP FOR condition too:

cBarText     = "Paste"
cBarDesc     = "Paste From Clipboard"
mBarAction  = "SYS(1500, '_MED_PASTE', '_MEDIT')"
mSkipFor     = "EMPTY( _ClipText )"

 

The code to generate the tables is in the attached zip file as GenBars.prg. You can use this file as the template to set up your own version of this little utility, and Figure 1 shows how the data looks for the result:

 

Now all we need is the code to use this. We created this as a class that does all the work in its INIT() event so that it runs itself when called. By returning .F. on completion of the code we prevent the object from actually being instantiated so that there is no lasting impact on the system at run time. The class is based on the session base class and this means that it creates a transient data session of its own – again, to avoid any environmental impact.

However, this also means that the class cannot be defined visually and so, if you really wanted to have this object defined as a visual class you could have to use either a Toolbar, Form or FormSet – these being the three visual classes that can create a Private Datasession. (Note: Of the three, the formset is actually the one with the smallest memory footprint – so this may be the only time you might actually use a formset in VFP. Personally I don't care if the definition is visual or not and so I just use the Session base class).

The Init() method is very simple indeed, as follows:

PROCEDURE INIT( tcMenuName )
  LOCAL lcScript
  *** Have we got this menu definition
  IF This.GetMenuDef( tcMenuName )
    lcScript = This.BuildMenu( tcMenuName )
    EXECSCRIPT( lcScript )
  ENDIF
  RETURN .F.
ENDPROC

If the passed in menu name is found by GetMenuDef(), the BuildMenu() method is called. This creates a temporary MPR file using the data from the metadata tables and the file is then executed to display the menu. O)n completion of the menu action, the method returns false preventing the object from actually instantiating.

The GetMenuDef() method simply executes a SQL query, creating a cursor that contains the relevant data to generate the required menu:

LOCAL lcMenuName
lcMenuName = UPPER( ALLTRIM( tcMenuName ))
*** Populate the cursor
SELECT PB.cbartext, PB.mbaraction, PB.mbarskip, PL.ilnkseq ;
   FROM popnames PN, popbars PB, poplink PL ;
 WHERE PB.ibarpk = PL.ilnkbarfk ;
      AND PL.ilnknamfk = PN.imenupk ;
      AND UPPER( PN.cmenuname ) = lcMenuName ;
      AND NOT DELETED( 'poplink' ) ;
     INTO CURSOR curMenu ;
   ORDER BY PL.ilnkseq
 *** Did we get anything?
RETURN (_TALLY > 0)

This cursor is then used by the subordinate methods (GetBars() and GetActions()) that are called by the BuildMenu() method. Here is the script generated for the EditOptions pop-up and, as you can see it is a perfectly standard MPR file that uses ExecScript in the ON SELECTION clause to execute whatever action has been defined:

DEFINE POPUP editoptions SHORTCUT RELATIVE FROM MROW(),MCOL()
DEFINE BAR 1 OF editoptions PROMPT [Copy]
DEFINE BAR 2 OF editoptions PROMPT [\-]
DEFINE BAR 3 OF editoptions PROMPT [Paste]SKIP FOR EMPTY( _ClipText )
DEFINE BAR 4 OF editoptions PROMPT [\-]
DEFINE BAR 5 OF editoptions PROMPT [Cut]
ON SELECTION BAR 1 OF editoptions EXECSCRIPT( [SYS(1500, '_MED_COPY', '_MEDIT')])
ON SELECTION BAR 3 OF editoptions EXECSCRIPT( [SYS(1500, '_MED_PASTE', '_MEDIT')])
ON SELECTION BAR 5 OF editoptions EXECSCRIPT( [SYS(1500, '_MED_CUT', '_MEDIT')])
ACTIVATE POPUP editoptions

So how do we use this class? The PopMenu.prg is in my VFP root directory and the class name is "xMenuPop". My VFP startup program, called from config.fpw, includes the following two lines:

*** Set Projects menu Hotkey and Run the Projects menu
ON KEY LABEL CTRL+F12 NEWOBJECT( 'xMenuPop', 'D:\vfp90\popmenu.prg', NULL, 'Projects' )
NEWOBJECT( 'xMenuPop', 'D:\vfp90\popmenu.prg', NULL, 'startup' )

The first assigns the projects menu to my CTRL+F12 key, and the second, which is the last line in startup program,  runs it immediately. Similarly to use the EditOptions menu we simply add one line of code to the right-click of any control that we want to invoke the menu from:

NEWOBJECT( 'xMenuPop', 'D:\vfp90\popmenu.prg', NULL, 'EditOptions' )

The class definition, the DBC, tables and the sample data generation program are all included in the downloadable zip file attached to this blog, I hope you find it as useful as we do.
posted by andykr | 0 Comments
Filed Under:
Attachment(s): PopMenu.zip

Our Vancouver Devteach Experience

You may have noticed the blog has been quiet for a couple of weekends. This is because Marcia and I have been out of town to attend the DevTeach conference in Vancouver. As regular readers of this blog will know, I have for many years extolled the virtues of conference attendance and encouraged people to ‘put their money where their mouths are’ and attend conferences.

On occasion it has been suggested that this was an easy position for me to take because I have been, for more than ten years, a speaker at conferences and haven’t actually had to put my hand in my own pocket to attend. The fallacy with that argument is, of course, that preparing a conference session takes many hours of effort – it’s not just a case of throwing a few slides together and dummying up a few code samples, there's a paper to be written to support it all, and many hours of rehearsal review and rework. My own estimate is that a typical 75 minute session requires 80 hours of work, and a 4 hour “pre-con” can take several weeks! So while there may be no direct cash payment, the cost in terms of lost time (and income) can be considerable.

What is the point of this? Well, as you may know Marcia and I decided last year that we were not going to speak at conferences any more and that SW Fox in October 2008 would be the last time we would submit sessions. As it happens, Marcia was not selected for that conference anyway – but either way it was our last as speakers. So this year we decided to attend DevTeach in Vancouver. We might have chosen the Montreal venue last December (it’s a lot closer for us) but we were in England visiting family and so couldn’t make it. Besides, neither of us had ever been to British Columbia and so we decided to add a couple of days up front of the conference and have a little vacation time in Vancouver too.

Why DevTeach? Well, while Marcia does have some existing clients still in VFP, all her new work is in .NET and I have been working almost exclusively in SQL Server for several years now anyway. Since admission to DevTeach (primarily .Net focused) also included admission to SQLTeach (SQL server focused) it was an obvious choice. Not cheap mind you, but obvious. It was also our first non-FoxPro conference for several years. The cash investment alone (fees, flights and hotel), for the two of us, ran close to $5,000.00 – not an inconsiderable sum in these tough economic times.

So, how did we fare? Well, I have to be honest and say that I did not really enjoy the experience. In the sessions that I attended, there were some good speakers who gave solid presentations and imparted a lot of good information. A definite plus there! However there were also some truly appalling speakers who, in my opinion should not have been allowed in front of a paying audience. I am not saying that they were not knowledgeable, or that their topics were uninteresting, but their presentation technique was so bad that it was painful.

No names, no pack drill, but I actually walked out of one session (from the front row where I had been sitting). The speaker began 15 minutes late (he kept waiting [hopefully] for ‘stragglers’), then spent the next 10 minutes on his personal life history and eventually began talking without agenda, structure or plan. After another five minutes I realized that I had no clue what he was talking about and, much worse, felt that he did not know either. What was worse he actually asked me later how I thought he had done!

OK, so not everyone is a natural public speaker in the Jim Booth, Ted Roche or (putting false modesty aside) even Andy Kramek, mode. However, there are simple rules that, when followed, will allow anyone to give a competent and respectable presentation and what I am talking about is people who failed to follow these basic rules. This, to my mind shows a cavalier disregard for their audience who have, after all, paid good money, and also invested their time to come and hear them.

So there were a couple of sessions that I thought were bad. Why am I whining, have I never seen a bad session before? Of course I have (and to be honest, I’ve probably given some in my time). That isn’t why I didn’t enjoy the experience. Taken overall, the technical content (from my SQL-oriented perspective) was at least ‘Good’ and verging on ‘Above Average’. But, as I have so often said, the technical content of the sessions is actually the least important part of any conference.

What is more important is the ability to mingle with like-minded people who share common problems and issues and exchange knowledge and views – i.e. Networking. This is the aspect in which I felt that DevTeach failed miserably. The question is, why? There could be several reasons, and I am not really competent to assess them (after all I wasn’t involved in planning the conference).

The first is that it was unquestionably a very “local” conference. Most of the attendees seemed to be either from the Vancouver area, or to have relations in the Vancouver area. Consequently at the end of sessions they went home. Result – no-one was left in the hotel and there was no real opportunity to network – people were either in sessions, or simply not there.

Second, the hotel layout made it a poor conference venue. The session rooms were split between two floors. Registration, the .Net and related Tracks, the Trade Show and breakfast/coffee between sessions were all on the third floor, while the Keynote, SQL tracks and lunch were on the second. Doesn’t make for a good mixing environment! The only bar was “L” shaped (and quite small) and even in the lobby there was really nowhere to sit and chat (a couple of sofas was about it). Taken all in all it is hard to imagine a less suitable venue for encouraging people to get together in groups.

Third, there was no real conference material. Upon registration we were offered a tote bag and a two page conference schedule and that was the sum total of everything. There was no binder describing the hotel, environs (where should we go for dinner outside the hotel?) and conference facilities (general timings, location of meals, where to look for notices etc) and, initially, not even paper and pens (though they did appear on Day 2 I believe). There was also no conference CD (though I did get an Email telling me that session materials would be downloadable from 6/18 – a week AFTER the conference ended but that the speakers slide decks were available for download immediately). Not really the kind of supporting material I expected or am used to.

Fourth communications in general were weak, in fact, on reflection, they were nonexistent! For example, someone actually complained, in conversation with Marcia on the afternoon of Day 2, that there was ‘not even lunch’  provided. They didn’t realize that lunch was being served on the floor below – but then how would they? There was no information provided at registration, no signs, and no announcements – in sessions or anywhere else - as to what facilities were provided, or where they were located. Worse, there was no indication outside the rooms telling you what sessions were going being held. A simple chart for each room with Topic, Speaker and Time would have been very helpful. I ended up in the wrong room twice and had to do a hurried exit and re-entry when I realized that the topic being presented wasn't what I thought it was going to be. Little things, but…

Fifth, the Keynote. Ah how we love those keynote presentations. In this case the keynote was a demo of Visual Studio Dot_Next. Of course this was of great interest (NOT!) to us SQL Server people. If you are going to combine two conferences, why not have two keynotes? After all the speaker was just one of the regular conference speakers – not an imported celebrity (as at Russ Swall’s “Essential…” conferences), or even an Industry (i.e. Microsoft) expert as at the German Devcon or SW Fox. Couldn't a SQL Server speaker have given a SQL Keynote?

Sixth, out of session activities. This was the last nail in the coffin for me – there were none! Oh yes, the Vancouver IT community hosted a ‘free beer’ party on the Monday night. Unfortunately it was not in the Conference Hotel, nor even close, and unless you were a local (oh, hang on, most attendees were!) it was not an easy task to figure it all out. The only information was in an Email – no announcements, no organized travel (how about sign-up sheets for organizing groups to share Taxi costs….?), no information or directions on how to get there at the conference desk (which was unmanned most of the time anyway). But then this was nothing to do with the conference, and was, obviously, aimed at the local residents.

Apart from this there was nothing. No “mixers”, no “Show and Tell” evening sessions, no “mitt bier” evening sessions, just an empty hotel bar and lobby.  For the first time in more than 10 years attending conferences I was back in my room by 8:00pm every night.

Seventh, there was no passion. I attended 15 sessions over three days and I didn’t see one where I felt that the speaker was really passionate about their topic. For those of you who have ever seen Cathy Poutney, or Doug Hennig, or Jim Booth, Marcia or I (to name but a few) give a session, you will know what I mean when I talk about speakers with passion. The impression I had was that, for the DevTeach speakers, it was ‘just another day at the office’. Most of them made it clear that the sessions were not new and the feeling I got was that it was all a bit of a chore for them.

Finally, one of the things that I have always hated was there, in spades! This conference, more than any other I can remember, suffered from the “Inaccessible Speaker” syndrome. I do not recall, in three days, seeing any speaker outside of a session room – unless they were traveling in packs as when waiting to go out to their speaker dinner.

Trying to go and speak to a speaker is, for most attendees, pretty intimidating. When you have to interrupt a group of speakers who are interacting with each other it is positively frightening. At lunch, what attendee is going to go and sit down at a table of 6 speakers earnestly engaged in high level discussions of (presumably) great weight and import? The problem was compounded at DevTeach because most of the speakers didn’t wear their name tags outside of sessions (so unless you already know them you don’t even know that they are speakers) and so the only the time the speakers were really visible was in sessions.

I know that Marcia and I (and the vast majority of the VFP Speakers, encouraged strongly, and in some cases even required, by VFP Conference Organizers) always made a conscious effort to be approachable, and accessible to attendees. We deliberately tried not to sit at “speaker tables” at lunch, and to ensure that we were always in the bar/lounge after sessions (an easy task for us Smile [:)]). Alas that culture does not seem to have carried over to DevTeach despite its original roots in the VFP community.

My conclusion? Well, as I said, technically the conference was adequate. Most presentations were competent, some were good and only a few were really poor. I did learn stuff; some of it will be directly useful to me, some is potentially useful and some just improved my general background knowledge and understanding. I have no complaints in this respect.

However, when you factor in the ancillary costs (travel time, lost work time, flight, accommodation and meal costs) the return on investment was very poor indeed. So while for a local resident it was probably cheap training, for us it was a very expensive, unstructured, training course with only moderate value. I left knowing no-one whom I did not know before I arrived (though it was certainly nice to see a few familiar faces among the speakers) and with the total number of business cards in my case the same as on the first day.

That was most the disappointing thing. The reality is that the whole conference felt like a commercial training course (where you sit in the classroom with a bunch of people you don’t know and with whom you have no real opportunity for communication and, at the end of the day, you all go your separate ways). From that perspective the training course is actually better value – at least there is a consistent and clear learning program.

Of course, this is just my opinion and others who were there may disagree with me but, as my readers will already know, I always try to tell it like I see it and this is how I saw it. I still believe that conferences can be great value, I just don’t think that DevTeach Vancouver was one of them.

So will we go to another DevTeach? Probably not. To another conference? Probably (right now I am not sure when or where, but I expect that we will) because despite this poor experience I still believe that GOOD conferences amply repay the investment you make in them. 

posted by andykr | 6 Comments
Filed Under: ,

The Cost/Time/Content Triangle

The Cost/Time/Content Triangle is a simple way of representing the rather complex (not to say confusing) interaction between the three key components of any IT project - Cost, Time and Scope.  It provides a mechanism for visualising the effect of changes in any parameter and provides managers with a tool for quickly assessing the impact (and hence the risk) of changes to any one component on the others. 

The Cost/Time/Content Triangle is a graphical triangle whose three sides each represent a measure for an element of the project. By setting an appropriate scale for each side we can define the limits for the three key elements of any project. This is probably easier followed with an example, so let's consider a simple project that, based on the defined scope (the 'Content' element) has been estimated as comprising 4 key parts totaling 20 weeks of work as follows:

  • Main Data Screens             12 Weeks  (55% of the Content)
  • System Admin Functions         3 Weeks  (15% of the Content)
  • Standard Reports               4 Weeks  (25% of the Content)
  • User Definable Configuration   1 Weeks  (  5% of the Content)

The percentages refer to the contribution each part makes to the whole project. Having got the Content, and Time elements we can calculate the Cost using our standard formula which, in this example is very simplistic and based on a standard Development Cost of $25 per man-hour + 25%, giving us a total cost of $25k:

20 * 40 hrs = 800hrs @ $25 = $20,000 + 25% = $25,000

(Note: this is the estimated internal cost of undertaking the project, not the price we intend to charge to the customer; that is a wholly different algorithm Smile [:)])

We can now construct our initial Cost/Time/Content Triangle -  using these values to generate the scales.  The objective is to get a balance so we want to get our initial estimate centered in the triangle. As you will recall from your Geometry classes you find the center of a triangle by drawing a line from each vertex to the mid-point of the opposing side. Where the three lines intersect is the center.   Where each baseline intersects the opposite side of the triangle we set the appropriate estimated value; i.e. Cost = £25k, Time = 20 Weeks and Scope = 100% as shown at Figure 1.

Scope is, of course, the main driving force behind any project and in overall, will largely dictate both the project time scale, and the cost.  After all, the more you have to do, the longer it will take, and the more it will cost!  However, not all elements of a project scope will have equal weight, and invariably the actual scope consists of two parts, the 'Must Be Done list' and the 'WIBNIF (Wouldn't It Be Nice If…) list".  The key to assessing scope is to determine what percentage of the project each element will account for.

Note that, thus far, our Content and Time scales are directly related (10% of the content = 10% of the time) which is what we would expect since our time estimate is based directly on the content. Cost is also directly related to content at this point because it too is derived solely from the content estimate and all we have considered to date is the "must-do" components. In fact, there were (as there always are) a couple WIBNIFs that we estimated as follows:

  • Ad-Hoc Query Screen          2 Weeks
  • Ad-Hoc Report Generator      3 Weeks

Moreover our internal assessment is that both the "User Definable Configuration" and the "Standard Reports" are really WIBNIFs too (the basic project could be done as "Phase 1" with just Screens and Admin functions) . So our project scope could drop as low as 70% of the base estimate, or go as high as 120%. These are, therefore the limits for our "Content" scale, which we define as a linear scale. Similarly our time, based on this content, could be as short as 14 weeks, and go out to as much as 25 weeks and we should add an appropriate time scale that covers this range, with some additional margin at each end (we'll see why later). Draw lines from the Content and Time vertices to each point on the relevant scale - this gives us the CTC Triangle shown in Figure 2:

To complete the triangle we need to add the Cost scale. To get this we need to calculate the cost of the project given various alternate elapsed time scales – remember that we have based the estimate on the defined scope. Changing the scope will give us a different cost for a given time scale. Table 1 shows how this works:

and using this as the basis for our time scale we get the final triangle (Figure 3):

All of our estimates relate to the original estimated scope and a time scale of 20 weeks, so now we can assess the impact of changes by simply reading off the appropriate scale. To ascertain the cost of doing the project in only 15 weeks, we do the following:

  • Draw a line from the Cost/Content vertex to the 15 Week point on the Time axis
  • Draw a line from the Time/Content vertex, through the intersection between our new line and the 100% Scope line and extend it to the cost axis to get the revised cost

The value is about £30K, which equates to £2k per week and is equivalent to 24 weeks at the standard rate - doing things quicker always costs more!

Similarly, we can estimate the reduction in cost if the project can be spread over a longer period. For example if we took 24 weeks for the project it cost only £22.5k.  Although this equates to reducing our weekly rate to about $940, we still only have 20 weeks worth of work to do and (hopefully) would be able to employ our developer gainfully elsewhere to make up the difference.

Now what about those WIBNIFs? If there are no other changes in the original scope, but our client also wants the Report Generator (estimated at 3 weeks work – or an extra 15% on the original scope) we can immediately see our options:

  • To preserve the original cost, the project will have to take 6 weeks longer, reducing our cost to $960 per week, but gaining an additional 3 weeks of time in which to carry out the extra work
  • To deliver on the original time scale, the cost will have to go up to just over $31,000. Not only is this increasing the amount of work to fit in, but it is also increasing our risk of failure
  • Propose a compromise. Increase the project time line by 4 weeks, and the cost to $28,000. This gives the a cost for the report generator of $3000 but gains us an extra week over estimate in which to do the additional work

In precisely the same way we can rapidly asses the effect of scope changes. If the decision is made that User configuration is not required after all, but that an Ad-Hoc Query screen is, the net result is to change the scope by +5%, the estimate can now be revised as either:

  • Deliverable in the same time at an extra cost of £1.5k
  • Delivered for the same cost, 2 weeks later

The triangle is really useful when you need, as in the example above, to cope with changes in scope (of the upward variety) at short notice and need to be sure that you can not only cover the costs, but can explain to the client why it is going to cost so much more. However, as you will no doubt have realized by now, the key to the triangle is the initial setting of the scales. This is only as good as the information which you use to generate your estimates, and the relationship which you determine between cost and time. 

Typically the cost relationship is not as simple as shown in this example – if we really wanted to increase the scope by 15% and deliver in the original time scale our poor developer would be working over 60 hours per week for five months (not a good scenario!). The reality is that we would need additional resources and that will change the cost relationship. Adding a developer is not simply a question of increasing the hourly billing rate, it also increases the  fixed costs; each developer needs equipment, Employment, Administration and Management Overhead costs all increase too. Similarly if the workload falls below 40 hours a week it is not always possible to re-deploy the resource without losing income.

The reality is that the relationship between cost and time is typically a stepped relation, rather than linear. In fact, getting this relationship right is one of the hardest parts of any project estimate, but once you have it, the CTC becomes a powerful tool for assessing risk and managing change.

 

Finding the length of a string in a specific font

One of the things that I often found myself doing when working in VFP was trying to decide how large a textbox had to be in order to accommodate the maximum length of the data that was permitted for its underlying source given a specific font and size. As we all know, unless you stick to non-proportional fonts, the number of pixels required to display a string can vary dramatically depending on the font.

For example, the string "Andy Kramek" actually requires 88 pixels when displayed in Courier New, 10 point font. However, change the font to Arial and you need only 79, unless you make it Bold in which case you need 85. But if you change the font to Verdana then you need 85 for the plain text and 96 for Bold.

Given the multiplicity of fonts available you could be forgiven for thinking that determining exactly how much room to allow would be a fairly normal thing to want to do and therefore could reasonably expect VFP would have native function that would tell you. Unfortunately, it doesn't. Now, at this point you may be thinking that this isn't really very important. After all, VFP sizes text boxes automatically using the underlying field definition when you drag a table field from the data environment, or project manager, on to a form.

Unfortunately the native sizing algorithm is not very accurate! For example, dragging a column defined as VARCHAR(50) to a form creates a textbox 357 pixels wide when the font is defined as Arial 9pt. However, the actual size required for a character string of 50 characters (using the average character width for this font) is only about 250 pixels! Even worse, the sizing has nothing to do with the font defined for the form and, even when you change the form's font in the underlying class, VFP simply carries on sizing controls for its default (Arial 9) font. Now that is a lot of wasted space on a form and, even worse, can be confusing to users because the size of the textbox is not a very reliable guide to the amount of text that it can hold.

We also have to consider the situation in which we need to concatenate data which is stored in individual fields together (to display a ‘full’ name for example, or a ‘City, State and Zip’ line for an address). VFP cannot help us here, and so the normal solution is to decide, empirically, how big to make the textbox. There are actually two issues involved here. The first is to determine the number of characters which we want the textbox to be able to handle and the second is to work out the amount of space that number of characters will need given a specific font setting.

Concatenating Fields

The default behavior of VFP when concatenating fields is to simply add together the defined lengths and create a new field whose width matches the total. Consider a set of name fields – let’s assume ‘FirstName’ is set up as 25 characters and ‘LastName’ as 40. Now VFP will always assume the worst case scenario when concatenating these fields and will generate a result field with a width of 65 characters (just run a SQL select that combines two character columns and you will see that this is indeed the case).

However, this does not represent the reality of the situation. Using the sample data that ships with VFP, we find that the average length of first names is actually 5.85 characters, and for last names it is only 6.83. So really we  could use a textbox capable of showing, say, 25 characters and be confident that we would handle the vast majority of cases. While it is impossible to be prescriptive about this sort of concatenation it is easy enough to set a target size that reflects the typical results.

So the number of characters is defined either by the underlying data source directly, or by some reasonable guess based on the required display.

But what about the font?

While we may be able to determine how many characters we want to display easily enough, as we have already seen, the size of the textbox that we need will depend upon the chosen font. In fact it is not only a question of the size of the font, but also of the style (e.g. Bold or Italic) and the font face. Even different fixed pitch fonts vary in their space requirement for individual characters.

Fortunately VFP does have functions, or more accurately a single, heavily overloaded, function to get this information. FONTMETRIC() can access twenty – yes, that’s right, 20 – different font attributes depending on the input parameter. I find it hard to remember the calling options for even one or two of these and even though IntelliSense makes it a little easier we still need to retrieve individual attributes one at a time. Since each returns the value for a single character, the whole process gets tedious, not to say messy, when you are trying to retrieve several attributes for the same string.

The SizeStr() Function

The solution is to create a simple wrapper function to determine the elements of the sizing that we are interested in. While we could grab all 20 attributes, in practice there are really only  three things we are interested in. The maximum length that the specified string could possibly require, the average and the exact lengths of the specified string. With this information we can make a sensible decision based on how typical our test string is, and how much variance we need to allow for in sizing our textbox. The SizeStr() function displays the results in a little modeless form (Figure 1) and returns the exact length of the specified string in the specified font.

 

Figure 1: The SizeStr() Result Screen (the function actually returns the Exact Width)

The exact length of a string is actually calculated as a two step process. First we use the Visual FoxPro TxtWidth() function to determine the number of “Average Character Equivalents” in the test string when the specified font is taken into account. This calculated value allows us to treat proportional fonts as if they were actually fixed and so simplify the ensuing calculation. Thus, while a string containing 5 letters will always return 5 with a fixed pitch font, a proportional font will give different results depending on both the font itself, and any additional styles that have been defined. Table 1 shows the results from TxtWidth() for the string “This is a Test String”.

Having determined how many average character equivalents we have in our text string we can simply multiply this number by the Average Character size (FontMetric(6)) to get the exact size which is returned by the function. To get the Maximum size for a string containing the specified number of characters we then multiply by the Maximum Character Size (FontMetric(7)). Finally we multiply the Average Character size by the actual number of characters in the test string to determine the average length of the string.

But what about the height?

Unfortunately (again) VFP does not help us to decide how high a textbox should be when we change its font. FontMetric() will give us the actual height of the character in a given font, but we also need to allow for border height when sizing a textbox. In SizeStr() I do this by adding a factor of 28.125% of the character height in the specified font. This value has no theoretical basis, but, by inspection appears to be about right! The resulting value is shown on the display form.

The function is attached to this article and can be downloaded below. It accepts the following parameters:

  • tuInStr        [Required]  The input string to be tested
  • tcFName     [Optional (defaults to "Arial")]  The name of the Font    
  • tnFSize       [Optional (defaults to 9 point)]  The font size (in points)       
  • tcFStyle      [Optional] Font Style codes (Bold, Italic, Underline etc)        

The function pops up the modeless window shown at Figure 1, and when the window is closed returns the exact length of the input string in pixels. However, the parameters displayed in the form are all saved to an object in the function and it is simple to suppress the form and return the object directly.

Example

lnLen = SizeStr( “This is a test string”, “Garamond”, 12, “BI” )
? lnLen       && Returns 131

Hopefully you will find this little function as useful as I do.

posted by andykr | 3 Comments
Filed Under:
Attachment(s): sizestr.zip

Modeling Lookup Tables

One topic that caught my eye on the Foxite forum recently was a discussion of how to implement Lookup Tables. Like many such questions it provoked strong opinions and definitive statements of the "Thou shalt ...." type (i.e. the type of "dogmatic logic" based on opinion, instead of reasoned argument). The question centers on whether it is better to use a single table for all lookup data (the so-called "One True Lookup Table [OTLT] approach) or to have separate tables for each different set of lookup data that is required. As with many such data The usually quoted conventional wisdom is that:

Developers like the idea of a single lookup table, but Database Administrators do not

and the reason why developers like them is that they simplify the creation of maintenance screens and allow generic code to be written to handle the population of controls in the user interface. Conversely the reasons why DBAs do not like them are that (according to an entry in "Systems Engineering and RDBMS" at http://decipherinfosys.wordpress.com/2007/02/01/otlt-one-true-lookup-table/ )

This is an OO design pushed into the database tier and poses many problems including :

  • Not being able to force foreign keys (so your data integrity is compromised right there),
  • Not being able to use the right data-types since everything has to be a string in order to accomodate all data-type values (this itself can give rise to bad data issues since a numeric(4,2) constraint cannot be enforced –> thus domain integrity is out of the door,
  • Because of (b), the length for the string column will be huge,
  • One has to then write complex SQL queries in order to retrieve the data and also take into account the type conversions that occur when you join this table with the other tables…implicit conversion is not allowed in all RDBMS and is not a good thing anyways.

However, as all too often with such didactic statements, this is based on a couple of assumptions; first, that all lookup data must be contained within a single table and, second, that all "lookup data" is the same and must all be handled the same way. Now, if those assumptions are really correct, then there is no question that using a single table for all lookup data is a bad idea (for all the reasons given). However, are they really valid?

The key to understanding this issue lies in one of the most basic principles of good database design - which is that "a database must model the business process that it supports". When viewed from this perspective it turns out that the assumptions on which the argument is based are incorrect. What we are taking about here is what I defined in my article "Designing A Database – Part 2" (see November 2008 on this web log) as Support Data. In that article I said the following:

Although support data cannot be derived from other data and therefore requires permanent storage, it differs from core data in that it is does not form part of the body of data upon which business applications operate directly. This includes all data that is used define how the system behaves (i.e. metadata and preference data), or how the data actually stored in the system is interpreted (i.e. look-up and reference data). The support data is a key element in the database in that it is primarily concerned with the translation of data into information – and this applies equally to metadata as it does to look-up or reference data.

Notice the highlighted section. The rules of data normalization require that we use Foreign Keys to reference values that would otherwise be repeated in a table and such values are usually referred to as "lookups" leading to the idea that they are all equivalent in some fashion. In reality that is not the case and there, as indicated, TWO types of data that have to be dealt with - "lookup" and "reference" data. In the previous article I did not expand on the distinction because it was not really relevant in that context. Let me do so now, because this distinction is, in my opinion, the key to the whole OTLT issue.

Lookup Data should, as the name implies, consist only of things that can be "looked up" or, to put it another way, consist of only one piece of significant information. So for a lookup, the foreign key directly substitutes for a denormalized value. Conversely, "Reference Data" may comprise many pieces of related information and a the foreign key is used to retrieve the set of information that is applicable. You may be thinking, about now, that I am making an artificial distinction, but believe me, I am not. Table 1 shows a few examples of "lookup keys" and how they map to the different categories.

Notice that for all keys that fall into the lookup category there is always just one piece of information – a description, and nothing else. While in the reference data category there is always more than one piece of information associated with the key. I can't speak for anyone else but myself, but to me it is obvious that there is no way that storing "Customer Credit Terms" in the same data structure as "Units of Measure" would make any sense. 

So the assumption that all lookup data is somehow the same is false to start with. For anything that falls into the Reference Data category, an appropriately designed table should always be used and I absolutely concur with the premise that doing anything else is poor design.

But what about the Lookup Data category? The fact is that all of this data is actually 'enumeration' data and that is probably a better term to use than "lookup" for this type of data. Reduced to basics it comes down to a Key/Value pair. Even so, if we were to use a single table for all such data we do run into the issues raised by opponents of the OTLT approach. Specifically, keys become difficult to define and either we have to enforce uniqueness on the "lookup key" or we have to introduce a 'Type' column to create compound key (i.e. Type + Value). Neither option is attractive and it would indeed complicate the SQL required if we had to adopt such a solution.

But hang on! We are talking about relational databases here (we wouldn't have foreign keys if we weren't)! So why are we constrained to use a SINGLE table? The short answer is that we are not and, for enumerations (i.e. that consisting only of code and description) we can use a PAIR of tables. The structure that I like to use is shown in Table 2.

Note that all of the issues disappear if we adopt this approach. Each table has its own (surrogate) primary key, and the relation between Type (header) and Enumeration (detail) is maintained by the foreign key in the detail table. There is no need to artificially restrict the key values in the detail table and a simple SQL inner join returns the appropriate set of enumerations. If there are specific sets of data that are used often by applications, we can define views for them and create either stored procedures, or functions, to handle specific requirements. The added columns on the detail table are for the convenience of the UI developer. The "Active" flag is used to indicate whether the value should be made available in the UI, and the "Default" flag indicates which value (if any) in the set should be used as the default in the User Interface. They do not have any significance, or impact, at the database level.

So my position is that reference data requires its own tables, but enumerations can be handled using a header-detail pair of tables. This does, however, raise another question. Should enumerations even be stored in the database? After all this data can be handled directly in code if all it is doing is providing expansion of key values to their associated description. The short answer is that they most definitely should be, and there are several reasons.

  • If enumerations are defined only in application code, then adding a new item to the selection always means recompiling, and re-distributing, the application. Admittedly if the new item requires special handling in some way, then its addition would still require code changes, but the addition of, say, a new contact type would probably not require any code change at all and can be handled by simply adding a row to a table and letting the code that accesses the enumeration tables return the new type.
  • Unless the enumeration is stored in the database, there is no way to decode the enumeration key values for reports or when exporting data to another application (e.g. an Excel spreadsheet for reporting or analysis) or to a data warehouse.
  • When multiple applications access the same database, there is not only the issue of ensuring that enumerations are available to each but also that of ensuring that they are synchronized and consistent. The scenario where one application interprets an enumeration key of 1234 as "Home Phone Number" while another defines it as  "President" does not bear thinking about!

So whether we are talking about an On Line Transaction Processing, a Management Information System, or a Data Warehouse, we always have to cater for two distinct types of "lookup data". The enumerations that are used solely to normalize values that would otherwise have to be repeated and the reference data that is used to manage various aspects of the application. They are totally different entities and should be handled in different ways.

So is there a case for the "One True Lookup Table"? Not in the way that it is usually defined. The trick is not to confuse reference data with enumerations and not try and shoe-horn reference data that has different attributes into a single common structure. However, a pair of tables can certainly be a much more efficient way to handle enumerations than having lots of separate, identical tables, each of which contains only a few static rows.

For example a CRM application I worked on recently has 40 rows in its lookup header table  (i.e. we would have required 40 tables if each type were handled separately) and a total of 303 entries in the detail table (there are 50 "States" and 42 "Providers", so the average number of 'rows' per type is only about 7). Can forty, identical, seven row tables really be more efficient than the header-detail pair? Hardly!

posted by andykr | 6 Comments
Filed Under:

Working with Vertical Tables

What is a "Vertical Table"?
The short answer is that a vertical table is one which does not store data elements in separate columns. Instead data is stored in rows, with each row containing a description of what is being stored (the “attribute”) and the actual value. In a normal table, the requirement to capture an additional attribute means adding an additional column to the table. The result is that the table gets wider (i.e. grows horizontally) as new attributes are added.

In the case of a vertical table, each row contains both the description of the attribute and its value. The result is that as additional attributes are added to the table it acquires more records (i.e. grows vertically) but never gets any wider. At its simplest, a vertical table consists of only two columns, the attribute descriptor and its value, hence the alternate name for this type of table; an ‘attribute/value pair’ table.

Whilst vertical tables can be used in many scenarios, there are two key criteria that, when satisfied, indicate that a vertical table is actually required. The first is that the data being modeled is infinitely extensible. The second is that the data related to its owning entity is always accessed as a complete set. These criteria are directly related to the advantages (and limitations!) of vertical tables and so merit detailed attention.

The term "infinitely extensible data" is simply a fancy way of saying that there are no specific rules about the data in question. One of the most commonly encountered examples of infinitely extensible data is Postal Addresses. This may sound odd, but if you think about it, there are no hard-and-fast rules for how postal addresses are defined. For example the US postal address for an apartment might look something like this:

123 Crossover Street, Apt 12A
Akorn, OH 44123

While an equivalent type of address in the UK might be:

12A Cowsfoot House
234 Pastures Walk
London
SX1 3GG

Analysis shows that each actually contain the same basic set of attributes (Apartment Number, Street Address, City and Postal Code) but the way in which they are defined is totally different. Thus the UK postal code includes the County (more accurately it defines a unique "postal district" which means that the county is implicit) while the US zip code requires the State to be defined explicitly. So what is usually done is to create a table with columns for each attribute and screens that allow data for each column to be entered. The result is usually something like this:

Which, of course, is perfectly OK - as long as the table was not defined with purely US coding in mind (i.e. 2-letter state codes and 5/9 digit zip code) but there is obviously no standard for the way address details are handled and, in practice, very few addresses really require all of these columns anyway. Now consider the same addresses as they might be stored in a vertical table:

The first thing to notice, which is a benefit of this approach, is that we can now name our "columns" much more appropriately for each address. Second, we have no wasted space in the table. Each address consists of as many rows as it actually needs. However, there are some issues that do not arise in a more conventional table.

First, even though we know which rows belong to which owner, we have no way of knowing in which order they should be presented because we do not have standard column names that can be mapped to a UI or report. So we need an additional column for each set of attributes to define how the attributes should be interpreted.

Second, our UI (or report) needs to be able to dynamically assign the appropriate number of rows and their descriptions because, while the address for owner 123 has 5 attributes, owner 125 has only 4 while owner 124 has 6 attributes. More importantly the attributes that were used differ between addresses.

So, you may be wondering, why don't we all use vertical tables like this for handling addresses? The answer is twofold. First, that while addresses considered globally are infinitely extensible in any one context they are not. After all, all US addresses are handled the same way, as are all UK addresses. So in an application context there is no real need for this kind of extensibility, and even when there is, there are other ways to handle it.

The second reason is related to a major limitation of vertical tables, that is, in turn, based in the second criterion for requiring one; i.e. that the data is always accessed as a complete set by owner. What this means, in practice, is that vertical tables are really useful when you need to save and restore sets of data, but are less useful when you need to anyalze that data.

For example, with a vertical table it is easy enough to get the address for owner 124:

SELECT attribute, value
  FROM address
 WHERE owner_fk = 124
ORDER BY sequence

but it is difficult to construct parameterized queries on such tables because there are no defined columns by which to filter results. The only option is to search for specific “attribute = value” combinations. 

Thus to find how many addresses we have in New York state the query would have to be written as:

SELECT COUNT( DISTINCT owner_fk )
  FROM address
 WHERE attribute = 'State'
   AND value = 'NY'

And this, of course, assumes that all New York addresses were actually entered using the "State" attribute (and not a "County" or "Province" attribute). What is worse, grouped queries that are easily handled by conventional tables are even impossible (like totaling sales figures by State for example).

So, if not for addresses, then what should we use vertical tables for?

There are a myriad of possible uses but, in general, I have found that you cannot simply implement a vertical table in isolation. The schema required, if only to implement the UI, is generally much more complex.

Here is an example for handling Educational and Professional Qualifications. The problem with qualifications is that they truly are infinitely extensible, even within a single context. There are absolutely no "rules" and no standards that are applicable.

For example the "standard" abbreviation in the US for a nurse is "RN" (Registered Nurse). However, in the UK "RN" indicates "Royal Navy" and the standard nursing qualification is "SRN" (State Registered Nurse). The problem is not just international either, what qualification indicates a medical "Doctor"? In the US around 30 are recognized (including PhD - Doctor of Philosophy) and to make matters even worse, even within this set there are conflicts – DO could be either "Doctor of Osteopathy" or "Doctor of Optometry" while DPM is either a "Doctor of Pediatric Medicine", or a "Doctor of Podiatric Medicine" (better be sure and get the right one to treat your problem)!

The situation is even more complicated when you take into account the fact that there are at least three different, and unrelated, types of qualification that people hold:

  • Educational: These document your academic achievements and include things like your college degree and area of specialization. These are static because once you have gained an educational qualification, you have it for life
  • Professional: These document your competencies and include things like medical qualification boards and memberships of professional associations. These tend to be static, but can change, especially if there are various grades of attainment or membership
  • Certification: These document things that you are allowed to do and include things like practice licenses or specialist skills. What differentiates them from professional qualifications is that they expire and have to be renewed periodically

Thus we could have three completely different sets of information which all relate to a individual's qualifications and they require completely different sets of data. Educational qualifications typically include an awarding body, subject, date and grade. However, a time-based certification will also need an expiry date and a professional qualification may well need a Licensing Body and Number.

Obviously this data meets the "infinitely extensible" criterion. However, unlike addresses it also meets the second criteria because we would always want to retrieve the data in sets. For example, license numbers, in isolation, are meaningless because we would always need to know the context for a license (what it is for, who granted it, when it was awarded, when it expires and so on). Similarly the fact that someone has taken a C# programming course is not much use unless we also know the level for the course (Beginner? Advanced?), when and where it was taken and whether they passed or failed.

In order to model this we will need to use a vertical table to hold the actual data that relates to a qualification to but we will need additional tables to describe how that data is to be interpreted.

In order to support this, we also need to define attributes, and this too is a little more complex, requiring three tables as described below:

The full model for handling qualifications is shown at Figure 1 below:

Notice, however,  that the attribute definition portion of the model is entirely generic and could be used defining any type of attributes. Consequently, for any data entity that needs attributes all you need is to define the attributes in this model and create the necessary "detail" table to link them to the owning entity. In fact, I usually define views to handle retrieving the attribute data so that the complexity of the model is hidden, thus to retrieve the generic set of attributes for an "address" I can query the view by name:
SELECT * FROM dbo.v_trans_attrib WHERE attgrp_nme = 'Address'
Returns:

Or I can use the attribute code directly and get the set of attributes for a Certification: 
SELECT * FROM dbo.v_trans_attrib WHERE attgrp_cde = 'CERT'
Returns:

By defining these views so that they include all relevant keys and codes it is a simple matter to have the UI generate the necessary data to retrieve the attribute sets, their associated lookup values and data. Since every table has a surrogate primary key, the update code is also very simple (and generic). 

OK, so vertical tables can work, but why bother? There are really two reasons. First, as I have tried to show, vertical tables can provide a flexible, and more importantly a generic, mechanism for handling totally disparate data that would otherwise require multiple tables with different columns. Second, and perhaps most important, vertical tables can handle changing data requirements without requiring changes to table structure. Adding a new attribute is merely a question of adding a new row.

They are not always applicable, but when you are dealing with data that is both extensible, and heavily set oriented, vertical tables offer a powerful and flexible way of handling the data. The additional planning and work involved in setting up a vertical table based model and creating the generic supporting mechanisms is, in my opinion anyway, amply repaid by the benefits achieved in terms of reduced maintenance and flexibility that they permit.

posted by andykr | 3 Comments
Filed Under:

Modeling Hierarchies (Part 2)

In the last article I gave the theoretical background to using mapping to model complex hierarchies. In this article I will work through an example that is intended to illustrate how the use of the mapping approach can solve some of the key issues associated with managing hierarchies in an application environment.

For the purpose of this example consider the position of a sales representative, named Amanda Barry, who works out of the Massillon plant for a Company which has plants in various parts of the country. She is actually a member of  three separate hierarchies simultaneously. These are:

  • Sales Team
  • Plant Organization
  • Customer

We’ll examine each in turn and model them accordingly

The Sales Team hierarchy

The company Sales Force is organized into teams which are based in geographic territories. The hierarchy describing this structure is shown at Figure 1, Note that these nodes represent not only logical groupings, but in some case are also actual positions within the company and have people associated with them. For example, Amanda is a member of the “Great Lakes” team, which has a Manager (Charlie Davis). But the Great Lakes team is considered a part of the “Northern Sub-Division” which is merely a convenient way of sub-dividing the four teams which comprise the Eastern Division . There is actually no real person associated with the “Northern Sub-Division” Node.

To model this structure will require a minimum of two tables, as described above, one for the Node Definitions, and one for the Mapping data (the rip table). Since we already know that we will want to define multiple hierarchies, we will add a third table to the structure, to allow us to categorize nodes. The structure we will use is at Figure 2:

Figure 2: Basic table structure for mapping hierarchies

To map the structure we require entries in these tables as shown at Table 2

Now we can map the structure to the structure table by tracing the paths. Once all of the paths have been defined the data can be added to the hierarchy (the Node PKs are shown below the descriptions in Figure 1). The actual data is at Table 2.

NOTE: We use the “*” character not only to separate, but also to terminate, the key references in the rip field. The reason is so that we can query this field later and avoid issues associated with partial matches on key references when querying the table using the LIKE operator on the rip field.
Consider the following pair of rip fields: “10*20*30” and “10*20*300”.
Clearly they are not referring to the same node but if we were to query where the value is LIKE “10*20*30%” we would get both. By using a termination character we avoid this problem.

The final task is to link our people to the appropriate node in the hierarchy. For the purpose of this illustration we are not concerned with how the Employee data is stored, but simply assume that somewhere there are tables that include a key value for each employee and the usual associated information (Name, Title, Date Joined etc). This data is represented by the “Person” table in Figure 3.

We use an allocation table to allow us to link a single employee to different nodes (we already know that Amanda is a member of three hierarchies, so she will have to be associated with at least three nodes, one in each hierarchy).

The crucial point to notice is that this allocation table links people to the structure table, not to the node!

The reason for this is simply that the node table is only a look-up which allows us to give a meaningful name to a node in the hierarchy. It has no significance whatsoever and indeed, the same name may be used in many places in different hierarchies (how many “Admin” departments are there in the world?). If we were to attach people directly to the node record we would require a separate node record for each individual occurrence of a node in a hierarchy, thereby making the relationship between a node and its structure one-to-one, forcing us back into a very rigid structure.

Now we can add the necessary records to the allocation table for the people that we know about so far, plus the Vice President in Charge of Sales Nationally (Jane Kingsland) and the head of the Eastern Division (Martin Niles) and the manager and a sales representative for the Mid-West team :

What does this give us?

It now allows us to create static views that query the hierarchy and return matching members. Here are the necessary definitions for two views. The first returns the rip field associated with a given node description:

IF EXISTS (SELECT * FROM sysobjects WHERE id = object_id('v_GetRip') and sysstat & 0xf = 2)
  DROP VIEW v_GetRip
GO

CREATE VIEW v_GetRip AS
 SELECT NN.node_pk, NN.node_dsc, RTRIM( RF.node_path) + '%' AS rip_field
   FROM node_stru RF 
        INNER JOIN node_def NN ON RF.node_fk = NN.node_pk

A simple query is now all that is needed to return the appropriate rip field mask that defines the starting point for data selection. The important thing to recognize is that this ‘mask’ allows to extract information from any level of the hierarchy without the need to know anything about the level or the relationships between the nodes. For example:

SELECT rip_field FROM v_getrip WHERE node_dsc = ‘Great Lakes’
RETURNS “1*2*4*9*%”

while

SELECT rip_field FROM v_getrip WHERE node_dsc LIKE ‘Mid%’
RETURNS “1*2*4*8*
%

This view can then be combined with another static view to return a list of people who are members of whatever node we wish to query by:

IF EXISTS (SELECT * FROM sysobjects WHERE id = object_id('v_GetMembers') and sysstat & 0xf = 2)
  DROP VIEW v_GetMembers
GO

CREATE VIEW v_GetMembers AS
SELECT DISTINCT VR.node_dsc keyval, ND.node_pk, ND.node_dsc,PN.title_dsc, PN.fname_nme, PN.lname_nme
  FROM v_GetRip VR
       INNER JOIN ( node_stru NS INNER JOIN node_def ND ON NS.node_fk = ND.node_pk )
            ON NS.node_path LIKE VR.rip_field
       LEFT OUTER JOIN node_psn_alloc AL ON AL.nodestru_fk = NS.nodestru_pk
       LEFT OUTER JOIN person PN ON PN.psn_pk = AL.psn_fk

Simple queries on this view return the list of people who qualify:

SELECT * FROM v_getmembers WHERE node_dsc = 'Great Lakes'
RETURNS
1 Great Lakes Sales Rep     Amanda Barry
2 Great Lakes Sales Manager Charlie       Davis

while

SELECT * FROM v_getmembers WHERE keyval = 'Mid-West'
RETURNS
Mid-West  8  Mid-West      Sales Manager Ellen  Franks
Mid-West  8  Mid-West      Sales Rep     George Harrison

and

SELECT * FROM v_getmembers WHERE node_dsc = 'Northern Sub-Division'
RETURNS
Northern Sub-Division      4      Northern Sub-Division      NULL   NULL   NULL
Northern Sub-Division      8      Mid-West      Sales Manager Ellen  Franks
Northern Sub-Division      8      Mid-West      Sales Rep     George Harrison
Northern Sub-Division      9      Great Lakes   Sales Manager Charlie       Davis
Northern Sub-Division      9      Great Lakes   Sales Rep     Amanda Barry

Any query relating to People is now a simple matter. Assuming that we have sales information related to the ID of the person who made the sale, we can get the total sales for any node in the hierarchy by simply summing the sales associated with the list of people who are related to that node:

SELECT SUM( sales)  FROM sales_data WHERE sales_person_id IN 
   (SELECT psn_pk FROM v_getmembers WHERE node_desc = ‘Great Lakes’ )

The same method can be used to extract the members of a Sales Team

SELECT name FROM emp_data WHERE person_pk IN

   (SELECT psn_pk FROM v_getmembers WHERE node_desc = ‘Great Lakes’ )

The Plant Hierarchy

The second hierarchy of which our notional person is a member is the “plant hierarchy”, part of which is  illustrated in Figure 4.  

We can simply add the data for this hierarchy to our existing tables (Table 4) in exactly the same way as for the Sales Organization detailed above. In this case the Node Category will be "2" and the various plants are added as new nodes – their structure being added to the Node_Stru table as before: 

Now we need to associate the Sales Representatives with the plants which they represent (Table 5). In this case only Amanda Barry (Massillon) and George Harrison (Columbus) are directly associated with plants, so only two new entries are required in the allocation table (obviously there would be other people associated with the plants hierarchy but the objective here is to focus only on the Sales side of the business): 

Our existing views, of course, will work just as well with this hierarchy as with the Sales hierarchy. Thus:

SELECT * FROM v_getmembers WHERE keyval LIKE 'North Ohio%'
RETURNS
North Ohio 15 North Ohio    NULL   NULL   NULL   NULL
North Ohio 17 Cleveland     NULL   NULL   NULL   NULL
North Ohio 18 Massillon     Sales Rep     Amanda Barry  1

 

Notice that we can now sum our sales by any node in either hierarchy using the same basic query. To get all the sales for all Ohio Plants the query is simply:

SELECT SUM( sales )  FROM sales_data WHERE sales_person_id IN 
     (SELECT psn_pk FROM v_getmembers WHERE keyval = ‘Ohio Plants’ )

and to get a list of representatives who are working out of any given plant, we use exactly the same query as we used to get a sales team:

SELECT SUM( sales )  FROM sales_data WHERE sales_person_id IN 
     (SELECT psn_pk FROM v_getmembers WHERE keyval = ‘Columbus’ )

 

The Customer Hierarchy

Adding a customer hierarchy is as simple as adding the plant hierarchy. Figure 5 shows a partial hierarchy for a large customer group and shows the entries for the Node and Node_Stru tables directly on the diagram:

 

One more thing that we need to do to accommodate this new hierarchy is to link our sales people to the appropriate node in the hierarchy. For the sake of illustration, we are mapping our Sales Reps to the lowest level of this hierarchy (Amanda Barry to 'NE Ohio [Node 24]' and George Harrison to SE Ohio[Node 25]), the Division Head to the Region level (Martin Niles to NE US [Node 22]) and our VP to National Level  

The net result is that the VP (Jane Kingsland) is now associated with two structures; first, she is attached to the organization at the National Sales node, and second, to the customer organization at the "Wendy's USA" node. Similarly other members participate in multiple structures, so that our friend Amanda is actually in three. She is associated with the sales tree at the 'Great Lakes' level, with plants at 'Massillon' and customers at 'SE Ohio'. However, no matter how many structures an entity is associated with, we still only ever have one record in the person table and one record in the node definition table!

Maintaining the structure

As can be seen, using the approach detailed here it is very easy to map, and to retrieve the data associated with, any number of hierarchies. By defining simple views to hide the complexity of the actual structure it is possible to support any desired user interface and even to dynamically add and delete entire hierarchies. The structures illustrated here show the bare minimum of information that is necessary to manage the data but do show the principles that underlie the approach. An obvious refinement to the Node Structure table is to add a “level” key, to permit data retrieval across all nodes that lie at a specific level of a hierarchy (for example, this would make implementing a TreeView to display the contents of a hierarchy very easy).

By modifying the node structure table so that it includes an additional foreign key for the parent node, and the level, the structure can be maintained using triggers that, when inserting a new record, generate the rip field and even level automatically based upon the rip field and level of the node to which the new record is being related (Figure 6).  

The resulting table, for all hierarchies is shown at Table 7:

 

 

The T-SQL code for the insert trigger is very simple:

IF OBJECT_ID('node_stru_ins') IS NOT NULL
  DROP TRIGGER node_stru_ins
GO
/*********************************************************************
TRIGGER.: node_stru_ins
Function: Insert Trigger for Structure table that automatically creates
........: the Rip Field when adding a new item of structural information
Called..: INSERT INTO node_stru ( node_fk, parent_node_fk )
               VALUES ( <emporg_pk>, <parent_pk> )
**********************************************************************/
CREATE TRIGGER node_stru_ins ON node_stru FOR INSERT
AS
BEGIN

  UPDATE TSU
    SET node_lvl = CASE WHEN TSU.parent_fk IS NULL THEN 1 ELSE PSU.node_lvl + 1 END,
        node_path = CASE WHEN TSU.parent_fk IS NULL
                         THEN '*' ELSE RTRIM(PSU.node_path) END
                          + CAST( TSU.node_fk AS VARCHAR( 10 ) ) + '*'
     FROM node_stru TSU
          INNER JOIN inserted INS ON INS.nodestru_pk = TSU.nodestru_pk
          LEFT OUTER JOIN node_stru PSU ON TSU.parent_fk = PSU.node_fk
END
GO

To return the data in a form suitable for display in a treeview we need a little more code, but not very much. Here is a stored procedure that accepts either the PK of a node, or the actual Node Description and uses the rip field to get the nodes, and sequence them correctly, so that they can be displayed.

IF OBJECT_ID('GetTree') IS NOT NULL
  DROP PROCEDURE GetTree
GO
/********************************************************************************
SP Name.: GetTree
Function: Retrieve a Tree from the specified root node
Author..: Andy Kramek
Syntax..: EXEC GetTree @vRoot
Params..: @vRoot Either Title/Location Name or PK for the node_def Table
********************************************************************************/
CREATE PROCEDURE GetTree @vRoot SQL_VARIANT  AS
SET NOCOUNT ON

/* Figure out what we got given here */
DECLARE @node_pk INT, @node_dsc VARCHAR(50)
SET  @node_pk = 0
IF @vRoot > 0
  /* Primary Key */
  SET  @node_pk = CAST( @vRoot AS INTEGER )
ELSE
  BEGIN
    /* Node Name */
    SET @node_dsc = LTRIM( RTRIM( CAST( @vRoot AS VARCHAR(50) )))
    /* Get the pk for the specified node */
    SELECT @node_pk = node_pk FROM node_def WHERE node_dsc = @node_dsc
  END

/* We now have a PK so get the rip field as a mask for the specified node */
DECLARE @Rip VARCHAR(1000), @Max INT
SELECT @Rip = RTRIM(node_path) + '%' FROM node_stru WHERE node_fk = @node_pk

/* Get all matching data, INCLUDING the root node, into a temporary table */
SELECT EO.node_dsc, ES.node_lvl, CAST( 0 AS bit ) haschild_flg,
       CAST( 0 AS bit ) isleaf_flg,  ES.node_path, EO.node_pk, ES.parent_fk, ES.nodestru_pk
  INTO #tempstru
  FROM node_def EO, node_stru ES
 WHERE EO.node_pk = ES.node_fk
   AND ES.node_path LIKE @Rip
   ORDER BY ES.node_lvl, ES.parent_fk, EO.node_dsc

/* Now we need to process the data to set the Child and Leaf flags */
IF @@ROWCOUNT > 0
  BEGIN
    /* Declare the variables we'll need for the cursor operation */
    DECLARE @urrip VARCHAR(1000), @nLevel INT, @HasCh BIT, @sLeaf BIT, @nodestru_pk INT
    /* And the ones for manipulation */
    DECLARE @LastKey INT, @lnCnt INT, @LeafKey INT

    /* Create a cursor for the specified fields */
    DECLARE curelems CURSOR FOR
      SELECT node_path, node_lvl, haschild_flg, isleaf_flg, nodestru_pk
        FROM #tempstru    

    /* Open the cursor, and get first 'record' */
    OPEN curelems
    FETCH NEXT FROM curelems INTO @urrip, @nLevel, @HasCh, @sLeaf, @nodestru_pk   

    /* Set the counter */
    SET @lnCnt = @@FETCH_STATUS
    /* Process all records */
    WHILE @lnCnt = 0
    BEGIN
      SET @LastKey = @nodestru_pk
      /* Does it have any children */
      IF EXISTS( SELECT node_fk FROM node_stru
                 WHERE (node_path LIKE @urrip + '%'
                   AND nodestru_pk <> @LastKey) )
        BEGIN
          /* It does have children! */
          UPDATE #tempstru SET haschild_flg = 1 WHERE nodestru_pk = @LastKey
          /* Now figure out which of the children is the last at that level*/
          SET @LeafKey = (SELECT TOP 1 nodestru_pk FROM #tempstru
                               WHERE (node_path LIKE @urrip + '%'
                                 AND nodestru_pk <> @LastKey )
                                 AND node_lvl = @nLevel + 1 
                               ORDER BY node_dsc DESC )
          IF @leafkey > 0
             UPDATE #tempstru SET isleaf_flg = 1 WHERE nodestru_pk = @Leafkey
        END
      /* Get the next 'record' */
      FETCH NEXT FROM curelems INTO @urrip, @nLevel, @HasCh, @sLeaf, @nodestru_pk
      SET @lnCnt = @@FETCH_STATUS
    END  

    /* Lose the cursor */
    CLOSE curelems
    DEALLOCATE curelems
  END
  /* Get the results and lose the temporary table */
  SELECT * FROM #tempstru ORDER BY node_lvl, parent_fk, node_dsc
  DROP TABLE #tempstru
RETURN
GO

Calling this procedure for "Ohio Plants" returns the data shown in Table 8. Note that the "Has Child" and "Is Leaf" flags are added dynamically. The first is used to indicate that a node has children, the second to indicate that the node is the last node at the current level. These are useful when generating a tree by parsing the data into an array or other structure used by the UI development tool. The PKs are included so that any modifications made can be sent back to the database by referencing the relevant primary key. 

Unlike the Left-Right numbering approach, changes to hierarchies modeled using a rip field affect only those records that are directly involved. There is no need to reconstruct the entire hierarchy whenever a node is added, deleted or moved. All that is necessary is to change any records that include the affected node, replacing the relevant portion of the rip field with the amended version. 

For example, if we were to disband the Southern Sub Division (see Figure 1) and move its “Central” sales team to the Northern Sub Division and the “South West” team to the Western Division, a total of three records in the Node Structure table are affected (Table 9). The process is simple:

  • Retrieve the ripfield mask for the node to be removed (OldStru)
  • Retrieve the mask for the node to which all child nodes are to be re-assigned (NewStru)
  • Update the structure table replacing all occurrences of "OldStru" with "NewStru"
  • Delete the one record where the rip field is exactly equal to "OldStru"

 

Obviously, any tables which included references to the deleted record would also have to be updated, but the key point here is that nothing else has to change. All data which previously aggregated to the “Southern Sub” group is now going to be aggregated into either the “Northern” or “Western” divisions.

Conclusion

The mapping approach described here provides the most flexible and easily maintained method of modeling hierarchical data of any type. By separating structural information from definition the retrieval of hierarchical information is simplified to the point where simple SQL can be used directly instead of requiring table queries, recursive “Tree-Walking” operations or the complex and inflexible left-right numbering which are the only viable alternatives.

NOTE: The sample SQL2005 database used in this article is attached as a zip file. Simply restore the database to your local SQL 2005 instance to see how all this works in practice.

posted by andykr | 5 Comments
Filed Under:
Attachment(s): ripdemo.zip

Modeling Hierarchies (Part 1)

As a computer users, we deal with hierarchies every day and working with relational data involves hierarchical relationships of varying degrees of complexity from the simple parent-child-grandchild (e.g. Customer -> Order -> Item) to very complex relationships (e.g. Bill of Materials).

So what, exactly, is a hierarchy

A hierarchy is a representation (usually shown graphically as a tree-diagram) of the way in which elements of data are related to one another. Each element is represented in the hierarchy by a ‘Node’ (usually represented as boxes), and each node is related to one, or more, other nodes by ‘Edges’ (usually represented as lines). The way in which nodes, and their edges, is interpreted depends upon the information that the hierarchy is describing. Thus in organizational hierarchies the nodes may represent either “Positions” or “Departments”, and the edges then represent either a  reports to’, or an ‘is a member of’ relationship. Conversely in a Bill of Materials hierarchy the nodes typically represent ‘Assembly Units’ and the edges define ‘is made of’ relationships.

The position of a node is defined by the number of incoming links (its ‘indegree’) that define its ‘parents’ and the number of outgoing links (its ‘outdegree’) that define its ‘children’. A node with an indegree of zero and an outdegree greater than zero defines the starting point of a hierarchy and is referred to as a ‘Root’ node. Conversely a node with an indegree greater than zero and an outdegree of zero defines the end point of a hierarchy and is referred to as a ‘leaf’ node. Nodes where both indegree and outdegree are greater than zero are ‘intermediate’ and those where both are zero are ‘isolated’.

Types of hierarchy

Simple hierarchies are comprised of a single family of nodes that are directly related to each other. In this case the hierarchy is represented as a single ‘Tree’. More complex hierarchies may comprise more than one family of nodes and the hierarchy is a collection of ‘SubTrees’. However complex the hierarchy, it is a defining characteristic that it is possible to trace a path from any node in the hierarchy to any other node by following the edges. Another characteristic is that all nodes except the root have an indegree of at least 1. While there are an infinite variety of possible hierarchies that can be defined using Nodes and Edges, there are two basic classes of hierarchy; either ‘Symmetric’ or ‘Asymmetric’.

Symmetric Hierarchies

A symmetric hierarchy is one in which the number of levels is the same for all component subtrees (Figure 1). Notice that the subtrees are not themselves required to be symmetrical with each other, what defines the hierarchy as symmetric is that there is at least one node at each separate level of every subtree.

Such a hierarchy can be modeled using simple relational tables, where one table is used to represent each level of the hierarchyCC (Figure 2). The relationship between records in any pair of tables is always at least one-to-one and there can never be any gaps in the structure.

This is probably the commonest type of hierarchy and handles many scenarios including the both functional (e.g. Customers and Orders) and structural (e.g. Company Organization). However, it is constrained by the limitation that there can never be any gaps in levels of the hierarchy and, if you need to model the case where Level 0 always the direct parent of Level 1 data, but is sometimes the direct parent of Level 2 data, this approach will fail.

Asymmetric Hierarchies

An asymmetric hierarchy is one in which the number of levels is not the same for all component subtrees (Figure 3). Notice that, as with the symmetric hierarchy, the structure of the subtrees does not matter. The defining characteristic is that there is not always a node at every level of every subtree.

It is clear that this type of hierarchy cannot be directly represented by mapping the levels to a set of relational tables because there are gaps in the structure. The first subtree has no “Level 2” nodes, while the second has a leaf node at “Level 1” and nothing below it. The third subtree has both leaf nodes and intermediate nodes at “Level 2” and a Leaf node that exists at “Level 4” but is related directly to a “Level 2” node that also has children at “Level 3”.

Since there are no clear rules about how any given subtree is constructed, the only way in which such a hierarchy can be modeled is by defining the individual relationships that exist between pairs of nodes. Typically this is done by defining a reflexive (also known as ‘self-referential’) table in which each record defines a node and relates specifically to its immediate parent (Figure 4).

However, it must be noted that this approach relies on the fact that any given node can have one, and only one, immediate parent. If a single node can be linked to more than one parent (as is often the case in reality) even this approach fails. The only solution in such cases (often used!) is to break normalization and duplicate the node record for each unique parent.

Working with hierarchies

As stated at the beginning of this paper, the purpose of any hierarchy is to describe how elements of data are related to one another. The reason for doing this is to permit the aggregation of raw data so as to avoid the necessity of working directly with the details. Hierarchies allow us to define meaningful levels of detail, whether these are "Sales Teams" or "Component Assemblies" or "Project Plans" depends only on the interpretation. As long as we are dealing only with symmetric hierarchies there is really no difficulty. SQL allows us to perform the relevant aggregation operations on the series of related tables that describe the hierarchy directly. Thus to retrieve a list of all the nodes at any given level we simply query the relevant table. To find all children of any given node requires only a single join. To traverse the entire hierarchy requires a series of joins but is still directly possible using SQL.

However, as soon as we have to consider the possibility of an asymmetric hierarchy we run into a problem. As noted above, such hierarchies can only be modeled by defining the relationships between pairs of nodes directly. In order to reconstruct the hierarchy we have to perform an operation often referred to as ‘tree-walk’. In other words we have to carry out the following steps:

  • [1] Retrieve the starting key for the root node
  • [2] Find all records where that key is defined as the parent (“Set1”).
  • [3] For the first record in the result set find all records where that key is defined as the parent
  • [4] Repeat [3] until there are no more children
  • [5] Repeat steps [3] and [4] for each record in Set1

Unfortunately basic SQL (which is primarily a set based language) does not directly support such recursive operations, or functions that rely on them. (Interestingly such support was proposed for the SQL3 standard – the so-called ‘RECURSIVE UNION’ operator but it was removed in 1996 and there are no plans to re-introduce it into the standard). Recent versions of some database management systems (including SQL Server 2005) have included enhanced functionality to address some of these issues, but even so it remains problematic. There are basically three alternatives to using SQL:

  • Handle the issue in code. This is the solution adopted most often; tools and applications that rely on hierarchies typically include functionality that will perform a tree-walk transparently. The classic example is the "Bill of Materials Explosion" found in manufacturing and stock control applications. The problem is that such code is difficult to write and is, particularly when dealing with large volumes of data and many levels, comparatively slow to execute.
  • Use a nested set approach. This approach relies upon the fact that a hierarchy can be represented not only as a tree, but also as a nested set of data. The problem with this approach is that it is difficult to handle dynamic, or changing, structures. This approach is described in detail in the following section.
  • Use a mapping approach. This approach relies on the fact that it is possible to create, for every node, a detailed set of the paths which define its location. Such paths are used to define all possible results from ‘walking the tree’ for every node in the hierarchy. This approach is described in detail below.

The Nested Set approach

The nested set model relies upon the fact that instead of being viewed as a tree, a hierarchy can be seen as a set of nested data points. Figure 5 depicts the same hierarchy illustrated in Figure 3 with each node identified by a unique key (a single letter in this case).

This same hierarchy can also be viewed as a nested set by replacing the lines usually used to represent the edges of the hierarchy with nodes that are nested inside each other (Figure 6).

An examination of this diagram shows that it is, in fact, identical to the tree shown at Figure 5. Thus nodes “B”, “C” and “D” are children of “A”. Nodes “G” and “H” are children of “B” and “K” is the only child of “H”. This diagram can be represented in a data table by associating a pair of numbers with each node. These numbers defines the lowest number that exists within the given node’s subtree (if any) and the highest number within that subtree. Figure 7 shows the hierarchy with the first tree numbered.

(You may find it easier to visualize, if you think of the numbers as being defined by starting at the root node of the tree in figure 6 with the number “1” and then visiting each node in the tree in turn by following the edges and working Left-to-Right and Top-to-Bottom. As you enter each node, increment the number by 1 and assign it to the ‘left’ side of the node. When you reach a leaf node, increment the number by one and assign it to the ‘right’. As you return to each node after visiting all of it’s children, again increment the number by 1 and assign it to the ‘right’. This approach is sometimes referred to as ‘Left-Right Numbering’ for this very reason).

The result is that every node defines the limiting values that identify all of its children. It is then possible to retrieve all nodes that fall within a given subtree in a single query by selecting out those nodes whose ‘left’ value is greater than the parent left value and the ‘right’ value is less than that of the parent. This is obvious from figure 7 where Node B defines the range [2-9].  By selecting all nodes where “LEFT > 2 AND RIGHT < 9” we would get nodes “G”, “H” and “K”. This is a very powerful way of handling any type of hierarchy – whether symmetric or asymmetric and neatly gets around the issue of having to walk the tree dynamically by defining the limits as part of the node definition.

It is also easily handled within the context of the standard approach to defining the nodes in a hierarchy as a series of parent-child relationships. All that is needed is two additional columns in the table (usually named ‘lft’ and ‘rgt’ to avoid conflicting with the key words ‘LEFT’ and 'RIGHT’) to hold the limiting values for each node. Figure 8 shows such a structure and the data for the numbered portion of the hierarchy.

The real drawback to this approach is that it is really only suitable for structures that are static. Inserting or deleting a node, or moving  a node from one place to another, requires that the entire tree numbering be validated and re-calculated. This can be a lengthy and difficult process since it requires walking the entire tree in the specific order (Top-to-Bottom and Left-to-Right) to ensure that limits correctly reflect the dependent structures. This is not to say that it is not a viable approach, it is widely used when dealing with hierarchies that are essentially static and, as noted, it does get around the limitations of SQL by avoiding the need to use a recursive algorithm. However it is very rigid and is only the best solution in certain, very specialized, circumstances.

The Mapping approach

The third approach breaks with the traditional method of representing a hierarchy as a series of parent-child relationships defined in a single reflexive table. Instead of trying to represent the relationships between pairs of nodes, we describe the hierarchy as the set of all permissible paths between a series of isolated nodes.

In order to do so we simply traverse the hierarchy starting with the root node and note, for each node that we encounter, the identities of all the nodes, that lie between it and the root node. It is important to recognize that this methodology is firmly rooted in the definition of what constitutes a hierarchy – in that it relies on the principle that it is possible to trace an unbroken path between the root and any node. It is also important to recognize that because each node is treated individually, there are no internal dependencies to consider. It does not matter whether the hierarchy is symmetrical or asymmetrical. Any hierarchy can be described in this fashion.

Thus the hierarchy shown at figure 5 can be represented by the data shown in Table 1

The crucial element behind this concept is that instead of storing the data in a single table it is stored in a pair of related tables; the first defines the nodes themselves and the second defines the paths between them. The reason for separating the data into two tables is to allow for the possibility that the same node definition is used in more than one hierarchy. (For example in a hierarchy that links two organizations and defines the structure of each as a subtree, the same ‘department’ node may appear in each). Since there is no need to define the structure directly with the node we can normalize the data. Figure 9 shows the schema for the tables needed

Notice that the node definition table is now merely a look-up that allows us to assign a description to path. It is the structure table that now describes how the nodes are related and it is the path column in this table that is the key to the whole methodology. The path column can be used directly in SQL queries to retrieve sets of data that conform to a specific structural pattern and this allows us to rip data out of the associated tables. For this reason the ‘path’ field is usually referred to as a ‘Rip Field’ and the structure table as a ‘Rip Table

By attaching data to the structure, rather than to nodes, we achieve a degree of freedom that cannot be handled by any other approach. In the next article I will explore this approach in detail and show how it can be implemented.
posted by andykr | 2 Comments
Filed Under:

NODEFAULT, DODEFAULT() and Scope Resolution [::]

A recent thread on Foxite highlighted, once again, the confusion that surrounds this little group of commands and functions. The help file isn't exactly helpful either, so let's start with the definition of NODEFAULT from there:

NODEFAULT

Prevents Visual FoxPro from performing its default event or method processing for Visual FoxPro events and methods.

The first thing to notice is that this is a command and it has no options or parameters. It's job is to prevent the execution of the "invisible" code that exists in various native Events and Methods of the Visual FoxPro base classes. It is this invisible code that defines the intrinsic behavior of the various base classes. Unfortunately there is no documentation regarding which events and methods contain such code and that are, therefore, candidates for this command. However we can infer some examples, including Keypress (where the intrinsic behavior is to display keystrokes in the control), QueryUnload (where the intrinsic behavior is to release a form)   and GotFocus and LostFocus (where the intrinsic behavior is for the control to gain, or lose, focus).

The significance of this intrinsic behavior is that it always happens, irrespective of the number of intervening classes between the VFP base class and class on which an object is based and whether or not you have added custom code to a class, or instance. It is, therefore, an exception to the normal rule in which code added to a class or instance overrides code defined higher in the class hierarchy. However, it is also important to remember that it always executes after any custom code has executed. In other words not only does the intrinsic code always happen, it is always the last thing that happens.

Since this intrinsic code is what makes a control behave the way it should, why would you want to prevent it? The short answer is that while you rarely want to prevent it totally, there are situations in which you either want to prevent it under certain circumstances, or when you need it to occur at some specific point in your own code instead of at the end. For example, when using the Keypress event to trap a special key (like "CTRL+F5") you would not normally want the control to display anything and so you can use code like this in KeyPress to 'eat' the keystroke:
IF nKeyCode = 98
  WAIT "CTRL+F5 Pressed" WINDOW
  NODEFAULT
ENDIF
Another case is QueryUnload where you may want to prevent the form from releasing under certain conditions. Code like this will do the trick:
IF MESSAGEBOX( "Pending changes will be lost", 32+4, 'Close the Form?' ) = 7 && No
  NODEFAULT
ENDIF
One other thing to note about NODEFAULT is that it does not matter where in the code it occurs. Since the function is to disable the intrinsic behavior it is either executed (to disable the behavior) or not (to allow the behavior). In this sense it acts like a "flag" and so it can be placed anywhere in your custom method code.

So far, so good. Now let's take a look at DoDefault() as defined by the help file:

DODEFAULT( [ eParameter1 [, eParameter2] ...] )

Executes the parent class event or method of the same name from within a subclass.

Clearly this is very different. First it is a function (which means that it returns a value that, potentially, can be checked) and second, it takes an indeterminate number of parameters. It's job is to call the method that has the same name in the parent from code in a child object. Despite the help file reference to "from within a subclass", you can in fact use DODEFAULT() directly in an instance to call code that is defined in parent class of the object.

Since the purpose is to make a method call, we have to be able to pass any necessary parameters and retrieve any result that the method call would normally return and so DoDefault is implemented as a function instead of a command. The importance of DoDefault is that it allows us to call one level back in the class hierarchy to execute code that would, otherwise, be overridden by code added directly in a child object. It is this function that allows us to add code incrementally to sub classes so that additional behaviors can be achieved by using the appropriate sub class.

For example, the Deactivate() event of our base form class includes the following line of code:
ACTIVATE SCREEN
This means that, when working in development mode we can run forms that are modeless (so we can access the command window) and test values, or call methods and have the results displayed to the screen and not clutter up the form. However, in applications we occasionally need to add custom code to the Deactivate event of a form and that would, of course, override our base class code. So we use the DODEFAULT function and, since the Deactivate event takes no parameters and does not return a specific value (remember that, like all VFP methods, it does in fact return a value of .T. by default, but in this case we really don't care) we end up with code like this:
IF ThisForm.ResultsToScreen
  DODEFAULT()
ENDIF
By including a DODEFAULT in these situations we achieve two things. First we retain the behavior defined in our base class, instead of overriding it. Second, and more importantly, we maintain the inheritance chain. In this example we could simply have included the single line of code directly in our form – however that would override the base class with the consequence that if we subsequently changed the base class behavior in some way the  change would not be propagated to the form. By using DoDefault – even where it is just a single line of code that is to be executed – we maintain the ability of objects to inherit from their parent class.

Since DODEFAULT() is a function call it is executed immediately and so it does matter where it exists. Effectively it inserts any code defined in the parent class into the code of the currently executing class so, depending on your situation, you can call the parent class before the child code, after the child code, within the child code or even conditionally at any point.

Notice that this function has nothing to do with the base class code and the suppression of intrinsic behavior that is controlled by NODEFAULT. It is concerned solely with method code in the immediate parent class. If there is no code in the immediate parent class, then DODEFAULT() has no impact.

So why is there any confusion?

The confusion arises from a very specific set of circumstances which makes it appear that DODEFAULT() calls the VFP Base Class intrinsic code. To show this I created a class library that contains six textbox classes and modified the GotFocus() event in each as shown at table 1:

Notice that in each case the second level subclass contains the same code, a NODEFAULT followed by a DODEFAULT(). The behavior of each class is shown at figure 1:

In each case, the first level sub class (i.e. the direct sub class of the VFP Base Textbox class) gets focus irrespective of whether or not the GotFocus() event contains any code. This is the behavior we expect since, by default, the intrinsic code executes after custom code (if any) and is always executed. However the second level sub classes behave very differently indeed.

In each case, the NODEFAULT explicitly suppresses the execution of the base class code at that level and so, in the absence of the call to the DODEFAULT function, none of these textboxes could ever get focus. The result of the call depends directly on the content of the parent class method as follows:

  • In the first case, RootSub2, the DODEFAULT() call results in a call to an empty parent class method, which goes nowhere and so the second level NODEFAULT prevents the subclass from getting focus.
  • In the next case, DirectSub2,  the DODEFAULT() call results in a call to the parent class method which contains another DODEFAULT(). The result of this is, again, to call the immediate parent class method of the first level sub class. However, in this case the immediate parent is the VFP Base Class, so the code from the base class gets executed and the textbox gets focus despite the second level NODEFAULT.
  •  In the third case, CustomSub2, the DODEFAULT() call executes the custom code that displays the wait window from the parent class. However, there is no call to the DODEFAULT() function in the parent and so the base class code is not called and, despite showing the wait window, the second level NODEFAULT applies and the textbox does not get focus.

Remember, the effect of a DODEFAULT() is to "insert" the code from the parent class directly at the point at which the DODEFAULT() is placed. So in my little example we could re-define the code in the subclasses as follows:

RootSub2 = Suppress Intrinsic Behavior + Do nothing else

CustomSub2 =  Suppress Intrinsic Behavior + Execute Parent Class Code only

DirectSub2 = Suppress Intrinsic Behavior + Execute Parent Class Code that Executes the VFP Base Class Code

When viewed in this light, it should be clear that DODEFAULT() does not execute the base class code except in the very specific case when the immediate parent class is a VFP base class.

What about Scope Resolution Operator

Unfortunately this is another example of some rather poor help file content.

:: Scope Resolution Operator 

Runs a parent class method from within a subclass method.

A more accurate description for this operator would be that it allows a specific method to be called from a specific class in the inheritance hierarchy of the current object. This contrasts with DODEFAULT() which calls the same method in the immediate parent. The scope resolution operator allows you to call any method, from any class in the hierarchy, including the base class. Using the scope resolution operator we can effectively "jump" up the inheritance chain to any prior level without needing intermediate DODEFAULT() calls.

So in the case of our second level sub class textboxes in the previous section we could always force the execution of the base class code by simply calling the base class method directly like this:
Textbox::GotFocus()
This begs the question as to why we would need to do it at all. The answer is that in some cases we may want the base class to execute, but to execute before some custom code instead of after it. I have been using GotFocus() as the example for a very specific reason.

The default behavior of the VFP Base Class Textbox GotFocus() method is to set both the SelStart and SelLength properties to 0 – so the content is not selected on entry. By setting the native "SelectOnEntry" property additional code is run after the textbox gets focus to select the contents. However this only works when the user tabs into a textbox. If the user clicks into a textbox with the mouse, the contents are not selected.

It seems logical that, in order to create a textbox class that selects text on entry in all cases, all we need to do is to set the SelLength property so that any text in the control is highlighted upon firing GotFocus(), like this:
This.SelLength = 999
The snag is that, as noted above, the default behavior of GotFocus() is to set SelLength to 0 and, since the default code runs after any custom code the result is that this line of code is useless. What we need in this case is the ability to make our custom code run after the textbox has got focus – or to put it another way, to explicitly call the base class code before our custom code.

This is where the scope resolution operator comes in, because we can use it to explicitly call the base class method and then suppress the natural call using NODEFAULT. The following code does exactly that to create a textbox that will always select its content on entry and that respects the setting of the native SelectOnEntry property:
WITH This
  IF .SelectOnEntry
    TextBox::GotFocus()
    .SelStart = 0
    .SelLength = 999
    NODEFAULT
  ENDIF
ENDWITH
This code ensures that irrespective of where the user clicks, the entire contents of  the textbox are selected, whether this is a first level sub-class, an instance, or a higher level sub class.

posted by andykr | 3 Comments
Filed Under:

Controlling Controls in VFP (Part 2: CommandButtons)

In the last article I showed a methodology that enables controls to react to changes in a form level property by altering their appearance. In this article I want to look at another aspect of controlling how things happen in a VFP  application – specifically related to command buttons.

Many, but by no means all, developers adhere to the principle that functional code should never be placed in Events, only in methods and I am most certainly one of those developers. One of the most over-used Events in VFP is the Click() event of the command button. Here’s a very simple example from the _MiscButtons.vcx class library. There is a button class defined there named '_cmdOK' which has a caption of "OK" and the following code in its Click() method:
IF TYPE("THISFORM.PARENT") = 'O'
  THISFORMSET.Release
ELSE
  THISFORM.Release
ENDIF

So what's wrong with that? Well lets take a more detailed look, shall we. First, I have to say that I’m not sure that I’d necessarily expect an “OK” button to arbitrarily release the form on which it is sitting, let alone release an entire formset. Second, it is a highly dangerous piece of code because it is assuming an awful lot of things are being handled ‘somehow’. Third, and most important of all it is not really the responsibility of a command button to take action. The job of a command button is notify something that it has been clicked.

The last point is critical to understanding OOP and relates directly to the principle of encapsulation (which demands that each object be responsible for its own actions, and not rely on the implementation of another object for its own functionality).

In the case above, is it really the command button's job to decide whether to release the form or, if the form happens to part of a formset  to release everything! Of course it isn't. Apart from anything else there are usually things have to be done (like checking for pending changes or releasing external references) before releasing a form. Another issue is that if the 'OK' button does this, how does the user cancel? Does the 'Cancel' button also have this code in it's Click() event? What happens if the user tries to use the standard windows "close" button in the title bar, or chooses "File Close" from the menu?

What I am saying is that while we do need functional code to release a form that code should NOT be part of a command button and that leads me to the first critical point.

Functional code should always be placed in methods, not events. In other words, events should only contain the code to call the appropriate method, not the code itself, there are four reasons for doing this:

First, if you need to call the code from more than one place, the resulting call ends up being something like this:
ThisForm.PageFrame1.Page2.cmdOK.Click()

Which is not pretty, though not necessarily bad in itself. The real issue is that there is no way of knowing (when editing the code in the Click() event of the "cmdOK" button) that it is also being called from elsewhere. You run the risk of making a change in the functionality that would be perfectly reasonable when called by explicitly clicking the button, but that would totally inappropriate if called from elsewhere.

Now, it is true that the same applies to any method, but the point is that you are less likely to make an accidental inappropriate change in a method named "ReleaseForm()" than  in one named "Click()". This leads us to the second reason.

Code in Events is not self-documenting and so is harder to maintain. If the function of the "OK" button is to trigger the form shut-down process, then have the button call a method named "FormShutDown()" and put the necessary code there. This is directly related to the third reason.

Code in Events is hidden – by which I meant that you have no way of knowing where the code is located. If you have a method on a form named ‘GetNextItem()’ it’s a pretty safe bet that it won’t have code to release the form in it. But what is in the Click() of a button captioned “OK”? As we have already seen, it could be anything! Which brings us to the fourth and most important reason of all.

Methods should do one thing and one thing only. The shorter the method code the better because the more things that you put into a single method the less re-usable that method becomes. My recommendation, and personal practice, is always to create 'control' methods that call out to functional methods that perform specific actions. This is the entire code from the DoNew() method on my data management form class (which has a pageframe):
LOCAL loPage
*** Find the Active Page and call its save method
FOR EACH loPage IN Thisform.pgfDataForm.Pages
  IF loPage.PageOrder = Thisform.pgfDataForm.ActivePage
    loPage.DoNew()
    EXIT
  ENDIF
ENDFOR
Thisform.cMode = 'A'
Thisform.RefreshForm( .T. )

Notice that even this method does nothing more than call the "DoNew" method of the active page. This puts the responsibility for the action where it belongs (on the page) and means that the only code in the Click() event of the "New" command button is
ThisForm.DoNew()

Notice that it also ensures that the responsibility for setting the form's mode and calling the form-level refresh are placed correctly (i.e. on the Form) instead of inside a subordinate object like a button.

So, to get back to the point of this article. How, in practice, can we avoid putting code directly in the Click() of a command button? The first thing to do is to address the functionality of the command button itself.

As I said earlier the button has to respond to a click. This implies a bi-directional notification, first the owning application must be told “I have been clicked” but, equally important, the button should also inform the user that it has recognized their action. How often have you seen users keep clicking on a button because the application didn’t give them enough feedback?

So I modified by button root class and added a custom method named OnClick() (for the actual code) and a custom property named lDisableOnClick. The Click() event contains only the following code which never needs to change:
WITH This
  IF .lDisableOnClick
    .Enabled = .F.
  ENDIF
  .OnClick()
  IF .lDisableOnClick
    .Refresh()
  ENDIF
ENDWITH

As you can see this handles temporarily disabling the button (to be clicked it must have been enabled), calling out to the OnClick() method to do whatever is needed and then calling its own Refresh() to set its enabled state by evaluating its cObjMode property as described in the last article. The purpose of the lDisableOnClick property is merely to by-pass the disabling for those cases where a button is always enabled.

So far, so good. We now have a button class that conforms to the principle of not placing functional code in an event – but we haven't really addressed the issue of what code should be in the method, all we have done is to move the location of the code, after all OnClick() isn't really any more helpful as a method name than Click()!

However, a command button cannot have an independent existence. It must always be a part of some larger container. It may be directly on a form, or in a toolbar, or part of a composite control like a navigation bar. Consequently it must always have a ‘parent’ of some sort. One of the big advantages of Visual FoxPro’s implementation of OOP is its containership model that allows us to access an object’s parent through an exposed “Parent” property rather than having to maintain pointers.

This opens the possibility of implementing a Chain of Responsibility pattern to handle whatever action a command button wants to initiate. To do this we need to add a method to all container classes to which a command button could be added. This method, named "DoAction", takes, as a parameter the name of the action to call. It checks to see if the method exists on the container and, if not, simply passes the call on up to its own parent object. The code is entirely generic and is the same for all container classes except the Form:
LPARAMETERS tcMethod
LOCAL lcMethod, lcTxt, luRetVal

IF VARTYPE( tcMethod ) # "C" OR EMPTY( tcMethod )
  *** This is not a character string, so we have no choice
  *** Just pop a message and return what we got sent
  MESSAGEBOX( 'Parameter: ' + TRANSFORM( tcMethod ) + ' cannot be acted upon', 16, 'Unhandled Action')
  RETURN tcMethod
ENDIF

*** It's a string, but we need to find out if we have a method of this name
IF PEMSTATUS( This, tcMethod, 5 )
  *** Yep, so go ahead and execute it
  lcMethod = 'This.' + tcMethod + '()'
  luRetVal = &lcMethod
ELSE
  *** We can't handle it, so pass it on
  *** If the parent object has a DoAction, Call it
  IF PEMSTATUS( This.Parent, 'DoAction', 5 )
    luRetVal = This.Parent.DoAction( tcMethod )
  ELSE
    IF PEMSTATUS( ThisForm, 'DoAction', 5 )
      *** Try the Form
      luRetVal = ThisForm.DoAction( tcMethod )
    ELSE
      *** We can't handle it, and there's nowhere else to go
      lcTxt = "Sorry, but the function which has been requested is not available."
      lcTxt = lcTxt + CHR(13) + "Please contact IT to ascertain when it will be ready."
      MESSAGEBOX( lcTxt, 16, 'Under Construction')
      *** Return a NULL value from here
      luRetVal = NULL
    ENDIF
  ENDIF
ENDIF

*** And return whatever we ended up with
RETURN luRetVal

Note that this code will work in all situations – even one where neither the owning object nor the form has a DoAction() method. The only difference for the form class is that it is, by definition, the end of the chain and so its ELSE condition if it doesn't have the specified method is simply to display the message, like so:
LPARAMETERS tcMethod
LOCAL lcMethod, lcTxt, luRetVal

IF VARTYPE( tcMethod ) # "C" OR EMPTY( tcMethod )
  *** This is not a character string, so we have no choice
  *** Just pop a message and return what we got sent
  MESSAGEBOX( 'Parameter: ' + TRANSFORM( tcMethod ) + ' cannot be acted upon', 16, 'Unhandled Action')
  RETURN tcMethod
ENDIF

*** It's a string, but we need to find out if we have a method of this name
IF PEMSTATUS( This, tcMethod, 5 )
  *** Yep, so go ahead and execute it
  lcMethod = 'This.' + tcMethod + '()'
  luRetVal = &lcMethod
ELSE
  *** We can't handle it, and there's nowhere else to go
  lcTxt = "Sorry, but the function which has been requested is not available."
  lcTxt = lcTxt + CHR(13) + "Please contact IT to ascertain when it will be ready."
  MESSAGEBOX( lcTxt, 16, 'Under Construction')
  *** Return a NULL value from here
  luRetVal = NULL
ENDIF

*** And return whatever we ended up with
RETURN luRetVal

So, to summarize. By adding an OnClick() method and cAction property to our command button class we now have a button which never needs to have code added. All that is needed is to fill in the name of the method we want called when the button is clicked. By implementing a Chain of Responsibility pattern we don't even need to worry about whether the specified method exists or not, it will either be executed, or the error will be handled gracefully.

Just about now you may be wondering how I handle passing parameters to the method being called. The answer is that I don't! Remember the only function of the command button is to initiate the call. If the target method needs additional information (like a value from some other object) it is the responsibility of the method to get them. It is not the responsibility of the command button to pass them.

This, like everything in OOP comes down to understanding the responsibilities of an object and placing the code to handle the responsibilities correctly in the class and object hierarchies. For more details on this topic see other articles on this blog:

Designing and Building Classes
http://weblogs.foxite.com/andykramek/archive/2005/03/27/279.aspx

Design Patterns - The Chain of Responsibility
http://weblogs.foxite.com/andykramek/archive/2006/12/24/3060.aspx

 

posted by andykr | 0 Comments
Filed Under:

Controlling Controls in VFP (Part 1: Display Styles)

A question that often comes up is related to how to handle scenarios where controls on a form need to behave differently according to some condition. For example, in our applications our normal practice is to bring forms up in view–only mode. This means that the user has to take some positive action (click a toolbar button, or  a button on the form) to put the form into either Add (i.e. get a new blank record), or Edit (i.e. change the existing data). This requirement conveys several distinct benefits in the context of an application.

First, it guards against accidental data changes. If a form came up in edit mode, with the first field selected, an accidental press of the delete key would delete the data. In our scenario this can't happen because they form is read-only by default.

Second, it gives us an opportunity to determine whether the user has permission to carry out their intended action. As a general rule we don't like the 'magic disappearing button' approach, but we certainly want to be able to disable the "Add" and "Edit" buttons if the current user does not have anything other than "View-Only" permission.

Third, it allows us to control the functionality so that only the appropriate options are enabled. Thus when the form first appears (assuming the appropriate permissions) the only options enabled might be Add, Edit, Search, Print and Exit. Upon entering either Add or Edit mode we would want to disable all the previously enabled options and enable only the Save and Cancel options instead. Not only does this help the user know what they can do, it prevents them from navigating away from a half-finished entry and it obviates the need for testing to see if changes were made (and the need for the irritating "Do you want to save your changes?" message that appears in so many applications when the user hasn't actually made a change).

Fourth, it enables us to present a consistent and intuitive behavior for our interfaces. This is by no means an insignificant factor because the key to a good user interface is that it should be obvious to a user what they are supposed to do. After all, having a "Save" button enabled does not make much sense unless the user has actually done something that might need to be saved!

So the question is how can we do all this without writing lots of repetitive code?

The first thing we need is have some means of determining what mode a form is in at any given time. Fortunately this is simple enough as there really are only three options for this; View-Only, Edit (existing Data) or Add (new data). So all we need is, in our form class to add a property for the form mode. In our classes this is named "cMode" and takes a single letter, either "V", "E" or "A" and has a default value of "V".

 

NOTE: Since this is not a user-controllable value there is no absolute need to add checking or validation to the property, it will only ever be set in code. However, if you wanted to ensure that these values are the only ones ever applied, an assign method would be a great way to handle it. The following code does exactly this, defaulting to "V" if an invalid value is passed in.

cMode_Assign Method:
LPARAMETERS tuNewValue
*** Default to View if empty or not character otherwise get, and make upper case, the first letter
tuNewValue = IIF (VARTYPE( tuNewValue ) # "C" OR EMPTY( tuNewValue ), "V", UPPER(LEFT( tuNewValue,1)))

*** Default to "V" if not V, E or A
tuNewValue = IIF (INLIST( tuNewValue, "V", "E", "A" ), tuNewValue, "V" )

*** Now set the mode
This.cmode = tuNewValue
RETURN

Now that we have a form mode property all we need to do is to make our controls recognize it. The basic methodology is simple enough but the implementation is slightly different for different types of control. For editable controls (TextBoxes, Edit Boxes, Combos and Lists) we use colors to indicate the state of the control. For example a control that is read-only is displayed with black text on a light grey background, while a control where an entry is mandatory is displayed as black on yellow. (Figure 1).

 

Figure 1: Controls that give visual cues

In order to implement mode-aware controls we need to modify the root classes.  These are the first level sub-class from the VFP base class on which all your controls are based. By adding this functionality at the root level all sub classes will inherit it. We will need two properties, and a little method code. The properties are named "cObjMode" which is used to control the object's mode,  and "cModeExpr" which is used to define the rule for modifying the object mode at run time. The main code that we need goers into the Refresh() method of the class and, is slightly different for different types of control:

Mode-Aware TextBoxes, EditBoxes and Combos

Have the same functionality for Refresh() and are set up like this:

WITH THIS
  DO CASE
       CASE .cObjMode = 'O'  && Optional
          .BackColor = RGB( 255, 255, 255 )  && White
          .ForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 0  && default
          .TabStop = .T.
          .ReadOnly = .F.
       CASE .cObjMode = 'M'  && mandatory
          .BackColor = RGB( 255,255,0 )  && Yellow

          .ForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 0  && default
          .TabStop = .T.
          .ReadOnly = .F.
       CASE .cObjMode = 'S'  && system set
          .BackColor = RGB( 128,128,128 )  && Dark Grey
          .ForeColor = RGB( 255,255,255 )  && Black
          .MousePointer = 1  && arrow
       CASE .cObjMode = 'R'  && read only
          .BackColor = RGB(192,192,192)  && Grey
          .ForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 1  && arrow
          .TabStop = .F.
          .ReadOnly = .T.
       OTHERWISE     && Default to Optional
          .cObjMode = 'O'
          .BackColor = RGB( 255, 255, 255 )  && White
          .ForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 0  && default
          .TabStop = .T.
          .ReadOnly = .F.
       ENDCASE
ENDWITH

Notice that this code ensures that a Read-Only control is removed from the Tab Order, but is still selectable when needed. This contrasts with the "System Set" which is not even selectable. That is handle by a single line of code the When() event:

RETURN NOT ( This.cObjMode = "S" )

that effectively disables the control completely when the Object Mode = "S".

Mode-Aware Lists

The code here is almost identical, but we must specify the Item Foreground and Background colors:

WITH THIS
  DO CASE
       CASE .cObjMode = 'O'  && Optional
          .ItemBackColor = RGB( 255, 255, 255 )  && White
          .ItemForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 0  && default
          .TabStop = .T.
       CASE .cObjMode = 'M'  && mandatory
          .ItemBackColor = RGB( 255,255,0 )  && Yellow
          .ItemForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 0  && default
          .TabStop = .T.
       CASE .cObjMode = 'S'  && system set
          .ItemBackColor = RGB( 128,128,128 )  && Dark Grey
          .ItemForeColor = RGB( 255,255,255 )  && Black
          .MousePointer = 1  && arrow
       CASE .cObjMode = 'R'  && read only
          .ItemBackColor = RGB(192,192,192)  && Grey
          .ItemForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 1  && arrow
          .TabStop = .F.
       OTHERWISE     && Default to Optional

          .cObjMode = 'O'
          .ItemBackColor = RGB( 255, 255, 255 )  && White
          .ItemForeColor = RGB( 0,0,0 )  && Black
          .MousePointer = 0  && default
          .TabStop = .T.
       ENDCASE
ENDWITH

and the code in the When() is identical.

Mode-Aware Action Controls

CommandButtons, CommandGroups, OptionGroups, OptionButtons, Checkboxes, Pages (not pageframes) and Grids also get these properties. However, instead of appearance we use them to control the Enabled, Disabled and Visibility properties. Here is the code from the Refresh() of the command button class:

WITH This
    DO CASE
        CASE .cObjMode = "E"    && Enabled
            .Enabled = .T.
            .Visible = .T.
        CASE .cObjMode = "D"    && Disabled
            .Enabled = .F.
            .Visible = .T.
        CASE .cObjMode = "H"    && Hidden
            .Enabled = .F.
            .Visible = .F.
        OTHERWISE                && Default to Enabled
            .cObjMode = 'E'
            .Enabled = .T.
            .Visible = .T.
    ENDCASE
ENDWITH

Now we have our controls set up so that they have behavior built into them based on their individual mode setting. In order to implement that behavior we now need to go back to our Form class and add some code there too. The first thing to do is to create a decorator method for the form level Refresh() method. We do this by adding a couple of custom methods named "BeforeRefresh()", "AfterRefresh()" and "RefreshForm()". The first two, as the names imply, are template hook methods which, at the class level, are empty (and therefore return .T.).

The RefreshForm() method is the one that contains the code. It takes an optional parameter (that defaults to .F. if not passed) as follows:

LPARAMETERS tlSetObjMode
WITH Thisform
  .LockScreen = .T.
  IF .BeforeRefresh()
       IF tlSetObjMode
          .SetObjMode( This, .T. )
       ENDIF 
       .Refresh()
       .AfterRefresh()
  ENDIF
  .LockScreen = .F.
ENDWITH  

The first thing this method does is to manage the form's lockscreen property. Next it calls out to the Before Refresh() hook. If that does not return .T. then the lockscreen is released and nothing else happens. If BeforeRefresh() returns .T. the next thing depends on the input parameter. If it is .F. (or was omitted) the form level Refresh() is called immediately, otherwise an extra call to the form's 'SetObjMode()' method, passing a reference to the form itself and a logical .T. (to suppress immediate refreshes), is made first. As the name implies, this method is responsible for setting the cObjMode property on all controls on the form.

We use recursive code to handle this:

LPARAMETERS toObject, tlNoRefresh
LOCAL loRef, lnCnt, loPage, loColumn, lcExpr, luLastMode

DO CASE
  CASE toObject.BaseClass = 'Pageframe'
    *** Validate each page
    FOR EACH loPage IN toObject.Pages
      IF loPage.PageOrder = toObject.ActivePage
        *** Loop through the objects on the pages
        FOR EACH loRef IN loPage.Controls
          *** Validate form objects that require validation
          Thisform.SetObjMode( loref, tlNoRefresh )
        ENDFOR
        EXIT
      ENDIF
    ENDFOR

  CASE INLIST( toObject.BaseClass, 'Form', 'Container' )
    *** Loop through all the objects on the form or in the container
    FOR EACH loRef IN toObject.Controls
      *** Validate objects that require validation
      Thisform.SetObjMode( loref, tlNoRefresh )
    ENDFOR

  OTHERWISE
    *** Evaluate the Mode Expression and refresh the control's cObjMode
    IF PEMSTATUS( toObject, 'cModeExpr', 5 )
      IF ! EMPTY( toObject.cModeExpr )
        lcExpr = toObject.cModeExpr
        luLastMode = toObject.cObjMode
        toObject.cObjMode = &lcExpr
        *** Only refresh if value has changed
        IF ! tlNoRefresh AND toObject.cObjMode # luLastMode
          toObject.Refresh()
        ENDIF
      ENDIF
    ENDIF
ENDCASE 
RETURN

This method loops through the form's controls collection and for container classes calls itself, recursively. For any control that it finds that has a non-empty cModeExpr property it evaluates the expression and sets the cObjMode accordingly.

Now you may be wondering why we have the extra parameter to suppress the immediate refresh of the control. The reason is that this method could be called from other places than RefreshForm(). When called from RefreshForm(), the next thing will be always a full form-level refresh, so there is no point in immediately refreshing each control individually. However, when called directly from a control (when the value in some control that affects another control directly has changed for instance) there will be no need to call a full form-level refresh and so we only want to refresh affected controls.

When all is done the RefreshForm() method calls the native form level Refresh(), then the AfterRefresh() hook and finally explicitly releases the LockScreen that it placed.

We are now finished! All that we have to do to implement this functionality in a form is to set the initial values for the cObjMode property to whatever is appropriate and then add the logical expression to handle changes to the cModeExpr property like this, for a "Save" button which will enable itself in Add or Edit mode but be Disabled in View mode:

IIF( ThisForm.cMode = 'V', 'D', 'E' )

Or this for a textbox which is Mandatory in Add mode, Optional in Edit mode and Read-Only in View:

ICASE( ThisForm.cMode = 'V', 'R', ThisForm.cMode = 'E', 'O', 'M' )

Wherever you need to use the functionality simply replace calls to ThisForm.Refresh() with calls to ThisForm.RefreshForm(.T.). So in an "ADD" button we would use code like this:

APPEND BLANK
ThisForm.cMode = 'A'
ThisForm.RefreshForm(.T.)

While for an "EDIT" button the equivalent code would simply be:

ThisForm.cMode = 'E'
ThisForm.RefreshForm(.T.)

Using this methodology you never have to worry about enabling/disabling controls again. Each control looks after itself based on its current environment. There are many other ways in which this type of approach can be used and I would be interested to hear of any that you come up with.

 

posted by andykr | 7 Comments
Filed Under:

Getting to Grips with SQL (Complex Queries)

In the introduction to his book “SQL For Smarties” (Morgan Kaufman, ISBN 1-55860-576-2) Joe Celko makes the point that often simply changing the way you approach a difficult problem can often help in resolving it. He goes on to offer a number of tips for approaching complex SQL problems. I have based these hints on his suggestions:

  •  First define exactly what columns you want to see in the result set. This will be the basis of the SELECT clause of the query
  • Next list all the tables that are necessary to get the required columns. This will be the basis of the FROM clause of the query
  • Then try to phrase the problem in terms of sets, not rows. This will be the basis of the WHERE clause.

If you can’t get the wording correct, try reversing it to see if the underlying logic becomes clearer. For example, instead of  return customers with more than one order” try stating it as “having more than one order is the criterion for inclusion”. Another approach is to negate the criteria. It is often easier to define what is definitely not wanted than to define all possible combinations that may be required. We could also phrase the selection criteria above as “exclude all customers who have only ever placed one order”.

Once you have the bare bones of the SELECT, FROM and WHERE clauses you can set about the process of refining the query and honing it for optimal performance and efficiency. As with many programming tasks the key thing is to ensure that the result is correct. After all, having the fastest query in the world is of little value if it gives the wrong answer!

As a matter of course you should always consider the possible impact of NULL values and empty tables. This is especially important when using nested sub queries or derived tables because a null value in a critical expression, or an empty table in a join, could result in a nominally “correct” query failing to return the correct result. Debugging such logical errors is difficult to say the least and it is much better to avoid the necessity altogether.

Query Examples

The following examples illustrate the techniques discussed in the preceding articles and show how to apply them to  complex, but fairly typical questions.

[1] What haven’t they ordered recently?

This is a very common question in any Sales organization. While it is easy enough to pull a customer’s last order and inspect it to see what they bought, it is a little tougher to find out what products they have bought in the past, but that were not on the last order that was placed. It is even harder to figure out how long it has been since they last ordered each product and yet this is exactly the kind of information that a salesman requires in order to call a customer.

Applying the principles outlined above, we’ll try and construct the query. Figure 1 shows the structure of the relevant portion of the database:

First we need to decide what fields we want in the result set and hence which tables we need to use. Clearly we need the customer name, the product name and the date it was last ordered (i.e. the maximum order date that includes the product). This gives us the SELECT clause of our query as follows:
SELECT CUS.cust_name, PRD.prod_name, MAX( TTOD( ORH.ord_date )) AS last_ordered
In order to get these columns we obviously need all four of the tables (even though we don't actually want any columns from the OrderDet table we need it to resolve the, otherwise many-to-many, relationship between OrderHdr and Products). The joins are quite straightforward in this case because we are only interested customers that have actually placed orders – so simple inner joins are all that we need, as follows:

FROM customer CUS
     INNER JOIN orderhdr ORH ON ORH.cust_fk = CUS.cust_pk
     INNER JOIN orderdet ORD ON ORD.order_fk = ORH.order_pk
     INNER JOIN prodmain PRD ON ORD.prod_fk = PRD.prod_pk

The filter is the tricky part of this query. Remember, the problem is to get all products that have been ordered at some time in the past, but which were not on the customer’s last order. However, by re-phrasing the question in terms of 'sets', the solution becomes clear:

Get the set of products and the date that each was last purchased by a customer, that were not part of the set of products on their last order

The first thing we need to establish is the date of each customer's last order – that's simple enough…

SELECT CUS.cust_pk cust_fk, MAX( ORD.ord_date ) lastorder

  FROM customer CUS

       INNER JOIN orderhdr ORD ON ORD.cust_fk = CUS.cust_pk

   GROUP BY cust_pk

We will use this query as a derived table (aliased as LOD for “last order date”) and join it to the customer table. So our from clause gets a little more complex:

FROM customer CUS
     INNER JOIN orderhdr ORH ON ORH.cust_fk = CUS.cust_pk
     INNER JOIN orderdet ORD ON ORD.order_fk = ORH.order_pk
     INNER JOIN prodmain PRD ON ORD.prod_fk = PRD.prod_pk
     INNER JOIN (
SELECT CUS.cust_pk cust_fk, MAX( ORD.ord_date ) lastorder
                    FROM customer CUS
                         INNER JOIN orderhdr ORD ON ORD.cust_fk = CUS.cust_pk
                   GROUP BY cust_pk ) LOD ON LOD.cust_fk = CUS.cus_pk

Now we need to add the filters that we want for this query. Obviously we need to filter out results where the order date matches the last order date and we would only want to do this for one customer at a time so we would also filter on the customer key. In this case we are using a variable named lnKey so that our WHERE clause becomes:

 WHERE CUS.cust_pk = lnKey
   AND ORH.ord_date < LOD.lastorder

Finally we must specify the grouping so that we can get the maximum (i.e. last) order date for each product and a sort order to finish off the query:

SELECT CUS.cust_name, PRD.prod_name, MAX( TTOD( ORH.ord_date )) AS last_ordered
FROM customer CUS
     INNER JOIN ( SELECT CUS.cust_pk cust_fk, MAX( ORD.ord_date ) lastorder
                    FROM customer CUS
                         INNER JOIN orderhdr ORD ON ORD.cust_fk = CUS.cust_pk
                   GROUP BY cust_pk ) LOD ON LOD.cust_fk = CUS.cust_pk
     INNER JOIN orderhdr ORH ON ORH.cust_fk = CUS.cust_pk
     INNER JOIN orderdet ORD ON ORD.order_fk = ORH.order_pk
     INNER JOIN prodmain PRD ON ORD.prod_fk = PRD.prod_pk
 WHERE CUS.cust_pk = lnKey
   AND ORH.ord_date < LOD.lastorder
 GROUP BY CUS.cust_name, PRD.prod_name 
 ORDER BY last_ordered DESC, prod_name

Which returns the following result set when the customer key is 77 (The Cracker Box)

As you can see, this customer last ordered “Laughing Lumberjack Lager” in December of 2003, but hasn’t ordered any Sauerkraut since July of that year. So maybe it’s time we found out what happened to their sales of sauerkraut?

[2] Who’s buying what?

Another very common problem related to sales operations is that of determining where customers who buy certain items, or groups of items are located. My sample data categorizes each product into one of  8 groups as follows:

Beverages (1), Condiments  (2), Confections (3), Dairy Products (4), Grains/Cereals (5), Meat/Poultry (6), Produce (7), Seafood (8)

To find out which customers in any given country are buying products from a particular category involves a complex series of joins, involving seven tables, as shown in figure 2:

However, applying the guidelines for approaching complex queries, we can see immediately that the only reason that we even need to consider the order and product related tables is so that we can determine which customers have purchased items from a particular product category. We do not actually require any data from these tables since the only thing we are interested in is the location of these customers. So if we now phrase the query slightly differently:

Find the names and addresses of all customers in any given country whose orders have included one (or more) products from any given category.

We can see that our basic query to get the customer information is going to be a fairly simple one involving only inner joins across the Customer, CustAddr and Address tables (Note: The custaddr table defines the type of address as either “Billing” or “Shipping” – in this sample of data all customers are of Type 1 (Billing) ):

SELECT CUS.cust_name, ADR.add_city, ADR.add_pcode
  FROM customer CUS
       INNER JOIN custaddr CAD ON CAD.cust_fk = CUS.cust_pk
       INNER JOIN address ADR ON CAD.add_fk = ADR.add_pk

Now, rather than trying to join the remaining tables, we can use an uncorrelated sub query to return a set of customer keys that match the specified criteria which, in this case, will be for some Product Category (lnPCat) :

SELECT ORH.cust_fk
  FROM orderhdr ORH
 WHERE ORH.order_pk IN
      ( SELECT ORD.order_fk
          FROM orderdet ORD
               INNER JOIN prodmain PRD ON PRD.prod_pk = ORD.prod_fk
         WHERE PRD.pcat_fk = lnPcat )

The key reason for taking this approach is that because it is an uncorrelated sub query it is only run once. This is one of those occasions on which a sub query may offer a significant performance benefit over using an embedded join - which must be evaluated for each record processed.

This gets us a list of customers who have bought products from the specified category. Now we have to filter out those who are not in the locale (country) in which we are interested – this will require another sub-query

To complete our SQL Statement we add this query, as a nested sub query and apply the final filter for the specified Country (lcCntry) and a sort order:
SELECT CUS.cust_name, ADR.add_city, ADR.add_pcode
  FROM customer CUS
       INNER JOIN custaddr CAD ON CAD.cust_fk = CUS.cust_pk
       INNER JOIN address ADR ON CAD.add_fk = ADR.add_pk
 WHERE CUS.cust_pk IN
       ( SELECT ORH.cust_fk
           FROM orderhdr ORHC
          WHERE ORH.order_pk IN
                ( SELECT ORD.order_fk
                    FROM orderdet ORD
                         INNER JOIN prodmain PRD ON PRD.prod_pk = ORD.prod_fk
                   WHERE PRD.pcat_fk = lnPcat ))
  AND ADR.add_cntry = lcCntry
  ORDER BY add_city

When this query is run for Product category = 2 (Confections) in the USA we get:

Of course this technique can be applied equally well to negative questions (who DIDN’T buy…) or to almost any conceivable combination of criteria. The ability to use nested sub queries in this way gives enormous flexibility.

posted by andykr | 2 Comments
Filed Under:

Getting to Grips with SQL (Sub-Queries)

A sub query, as the name implies, is simply a query that is contained within another query. The purpose of a sub query is to return one or more values to the outer query and there are three basic types of sub query:

  • Sub queries that generate a list of values (usually, but not always, keys) that meet a specified condition. Such queries must begin with an IN clause
  • Sub queries that generate a value that meets some criterion for comparison. Such queries must return a single value
  • Sub queries that test for the existence of a value. Such queries begin with an EXISTS clause

Sub queries that are valid (i.e. stand-alone) queries in their own right are referred to as "uncorrelated". For example the query:

SELECT prod_pk FROM prodmain WHERE pcat_fk = 7

is clearly perfectly valid in its own right and will return a list of keys for products that belong to the category 7 (which happens to be "produce" in my test data). However, it could also be incorporated, as an uncorrelated sub query, into an outer query which a list of the keys for that category for some reason ( like to get the set of customers who have included such products on an order). The following query lists, by customer and order date, the products and quantities ordered from the “Produce” category:

SELECT CU.cust_name, OH.ord_date, PR.prod_name, OD.odet_qty 
  FROM customer CU, orderhdr OH, orderdet OD, prodmain PR 
 WHERE CU.cust_pk = OH.cust_fk 
   AND OH.order_pk = OD.order_fk 
   AND OD.prod_fk = PR.prod_pk 
   AND OD.prod_fk IN ( SELECT prod_pk FROM prodmain WHERE pcat_fk = 7 ) 
 ORDER BY cust_name, ord_date DESC

It is characteristic of uncorrelated sub queries that they are executed only once. In the above example the sub query is executed first and its results are substituted into the WHERE clause of the main query. The result is, therefore, exactly the same as if we had written the query thus:

SELECT CU.cust_name, OH.ord_date, PR.prod_name, OD.odet_qty 
  FROM customer CU, orderhdr OH, orderdet OD, prodmain PR 
 WHERE CU.cust_pk = OH.cust_fk 
   AND OH.order_pk = OD.order_fk 
   AND OD.prod_fk = PR.prod_pk 
   AND OD.prod_fk IN ( 7, 14, 28, 51, 74 ) 
 ORDER BY cust_name, ord_date DESC

Except, of course, that we do not need to know the key values and, more importantly, if a new value were to be added the uncorrelated sub query would still return the correct results whereas the hard-coded version would return an incorrect result  (not an error, the result would still be valid, but it could be wrong) unless or until the code is modified.

However, there are many cases where we cannot simply pre-define the criteria for the sub query because we need to know the value of one or more columns in the tables used by the main query. Such sub queries are referred to as "correlated" sub queries because they are only valid in the context of the outer query. Correlated sub queries are executed once for each row returned by the outer query to which they relate and the result of the sub query is used to determine whether the row meets the specified condition.

The following example uses a correlated sub query to retrieve, in a single query, the details of the most recent order placed by each customer:

SELECT CU.cust_name, ord_date, ord_amt 
  FROM customer CU, orderhdr OH 
 WHERE CU.cust_pk = OH.cust_fk 
   AND ord_date = (SELECT MAX(ord_date) FROM orderhdr WHERE cust_fk = CU.cust_pk )
ORDER BY cust_name

In this case the sub query is clearly not valid in its own right because it depends upon the id from a table that is part of the main query. The result is, therefore, the same as if we had written a number of separate queries – each using its own uncorrelated sub query to find the most recent order date for a single customer, and aggregated the results:

SELECT CU.cust_name, ord_date, ord_amt 
  FROM customer CU, orderhdr OH 
 WHERE CU.cust_pk = OH.cust_fk 
   AND ord_date = (SELECT MAX(ord_date) FROM orderhdr WHERE cust_fk = 1 )
UNION
SELECT CU.cust_name, ord_date, ord_amt 
  FROM customer CU, orderhdr OH 
 WHERE CU.cust_pk = OH.cust_fk 
   AND ord_date = (SELECT MAX(ord_date) FROM orderhdr WHERE cust_fk = 2 )
UNION
SELECT CU.cust_name, ord_date, ord_amt 
  FROM customer CU, orderhdr OH 
 WHERE CU.cust_pk = OH.cust_fk 
     AND ord_date = (SELECT MAX(ord_date) FROM orderhdr WHERE cust_fk = 3 )

One obvious benefit of the correlated sub query is that we do not need to know, in advance, the id of each customer in order to write the query. However, there is also a performance benefit. Since a UNION is executed as a series of queries whose individual result sets then have to be aggregated it is a much less efficient method. In fact the correlated sub query required (in SQL Server) about 50 reads for the entire customer data set, while the union approach required approximately 30 reads per subquery. To obtain the results for three customers required about 90 reads, four customers 120 and so on (it is not actually a linear progression, but it is very close).

The differences between correlated, and uncorrelated sub queries can be summarized like this:

  • An uncorrelated sub query can be extracted from the main query and executed stand-alone. A correlated sub query cannot
  • An uncorrelated sub query is executed once for each query. A correlated sub query executes once for each row in the result set

Irrespective of their type, sub queries can be nested to further extend the capabilities of a single SQL statement. The main benefit of this is that it allows us to handle queries that would previously have required two or more separate queries as single query and, as illustrated by the last example in this paper, dramatically reduces the number of joins that we have to make.

Derived Tables and Computed Columns

So far we have only considered sub queries in the context of filters, however, sub queries are not limited to the WHERE clause of a SQL query. The intermediate result set that results from a sub query that is defined in the FROM clause of a query is known as a "Derived Table", while those from queries embedded in the SELECT clause are "Computed Columns".

Derived tables are useful when dealing with complex relationships because they make it possible to simplify the join conditions. The following query retrieves the phone numbers associated with each customer using a filtered outer join:

SELECT con_lname, con_fname, CC.ccon_det
  FROM contact CO 
       LEFT OUTER JOIN custcont CC 
            ON CC.cont_fk = CO.cont_pk AND CC.ctyp_fk = 1

By using a derived table we can move the filter into a sub query, and so have a simpler join to get the same result set:

SELECT con_lname, con_fname, PH.ccon_det
  FROM contact CO 
       LEFT OUTER JOIN (SELECT cont_fk, ccon_det FROM custcont WHERE ctyp_fk = 1) PH
            ON CO.cont_pk = PH.cont_fk

The common table expression, introduced in SQL Server expression provides an efficient alternative to using a derived table like this

A limited form of sub query is also permitted in the SELECT clause of a SQL query to generate a computed column. In this case the sub query must return a single value (either a valid result or the null value). This can be useful when we need a single aggregated value that would normally require using an Outer Join and Group BY cluse. Thus in order to get a value of the total sales by customer we would normally require a query like this:

SELECT CU.cust_name, SUM(OH.ord_amt) tsales 
  FROM customer CU 
       LEFT OUTER JOIN orderhdr OH ON cust_fk = CU.cust_pk
 GROUP BY cust_name

However, it is also possible to use a correlated sub query to generate the total sales as a computed column for each customer in the table. The actual query that retrieves the same results as the one above is:

 SELECT CU.cust_name, (SELECT SUM(ord_amt) FROM orderhdr WHERE cust_fk = CU.cust_pk) tSales
    FROM customer CU

Whether the computed column is more efficient than the more conventional approach will depend upon the complexity of the queries and the number and type of joins involved in resolving them but, as a general rule, a single correlated query is more efficient than using a group by clause – especially when the result set has more than one or two columns. However, multiple embedded computed columns will definitely slow a query down and should, as a general rule, be avoided. As always, the only way to be certain is to test!

Using EXISTS

As noted above, a very common use for sub queries is to build a list of keys that meet some criterion that are then used to select matching data. For example, the following query returns the list of customers who have never placed an order:

SELECT cust_name 
  FROM customer 
 WHERE cust_pk NOT IN (SELECT cust_fk FROM orderhdr) 

Notice that in this case we really don't care what the value of the customer id in OrderHdr table is, or how many times it is there. We only want to know if there is at least one occurrence of the id or not. In practice this scenario, in which we only want to determine whether some particular condition is met but don't actually care what the value is, or how many occurrences there are is very common. This is where we can use the EXISTS sub query.

The benefit of this form is that, irrespective of whether the query is correlated or uncorrelated, the sub query is only executed until a single matching result is found. In other words it effectively reduces the time required to execute the query by avoiding the necessity to process all rows. The following example uses a correlated sub query to return the identical record set as the one above:

SELECT CU.cust_name 
  FROM customer CU 
 WHERE NOT EXISTS (SELECT cust_fk FROM orderhdr WHERE cust_fk = CU.cust_pk)

NOTE: Although the focus of this paper is on queries, it should be remembered that, with certain limitations, sub queries can also be used in INSERT, UPDATE and DELETE statements. The exact form of sub query that is supported depends upon the nature of the command.

As always the best advice when in doubt as to whether a particular statement is valid is to try it!

 

posted by andykr | 4 Comments
Filed Under:

Getting to grips with SQL (The basics)

Looking back through the series of articles that I have published on my blog over the past couple of years I see that I have not specifically addressed the use of Structured Query Language (SQL). So it's time to remedy that omission. Before we can begin discussing the details of advanced SQL it is important to ensure that we understand some of the basic terminology that applies. There are a number of terms that are used when working with SQL so let's start with them.

Joins Conditions and Result Sets

A “Join Condition” is simply the set of criteria by which the records in a pair of tables are evaluated. Only records that meet the specified criteria for the join are included in the result set and all other records are ignored. A result set is simply that subset of the data that meets the specified join condition. It is important to remember that  although joins can only ever apply to pairs of tables, a single SQL statement may include several joins resulting in a complex query that spans multiple tables.

Note that while we are referring to joins as existing between pairs of tables, there is no reason why a table cannot be joined to itself. Such a join is sometimes referred to as a self-join but apart from the fact that it involves two copies of the same table there is no difference between a self join and any other type.

There are two forms of syntax when defining a join. The first, older, format uses the FROM and WHERE clauses of the SQL statement to specify the join. The second, ANSI 92 compliant (often referred to as the 'canonical’') join syntax defines the join explicitly and reserves the WHERE clause for filters, thus the same query could equally well be written in either of the following forms:

Whilst the two queries are functionally equivalent and both will work perfectly well in either SQL Server or Visual FoxPro it should be noted that the canonical join syntax is not supported by all versions of all databases (most notably some older versions of Oracle). So if portability of code is an important consideration it is generally better to stick to the first form. However, if you are working with Visual FoxPro, the canonical join syntax is the only way to define an outer join without resorting to a union.

Filters

A filter is an additional criterion that is used to refine a result set. Filters are applied after the join has been evaluated to further reduce the number of qualifying records. Often filters are based upon the contents of a single column (usually a key field) but there is no necessity to restrict them to a single column. The query above returns records for all contacts, but by applying a filter in addition to the join we can reduce the number of records in the result set to those of a specific customer.

Note that now the canonical join syntax clearly differentiates between the JOIN and the FILTER clauses whilst in the traditional SQL format there is no such clarity. For this reason alone the canonical join syntax is generally preferable where it is available.

Joins and their Type

As mentioned above, SQL relies on joining tables in pairs. In other words only two tables can participate in a join at any time. In the case of multiple joins, spanning many tables, the joins are evaluated sequentially with the result of joining the first pair of table acting as the the input table for the next join in the sequence. Thus a query involving three tables like this:

is actually executed by generating an intermediate result set from the join between the first pair of tables (ProdCat and ProdMain) and then joining the third table (OrderDet) to the generated intermediate result set. Different types of join produce different result sets as described below.

Inner Join

An inner join matches records in two tables according to the specified criteria. Only records where the specified criteria are met for records that exist in both tables are included in the result set. (Figure 1)

This is the join that is used when the word JOIN is used in a query without other qualification and so it is sometimes referred to as a “Natural” join. Thus “join” = “inner join” and the following queries are identical:


Outer Join

An outer join matches records in two tables, but preserves all records from the table on one side of the join whether or not there are matching records in the other table. The result set of an outer join therefore includes either data from the table that is not being preserved if matching records exist, or NULL values where they do not. Outer joins are usually defined as either LEFT, or RIGHT,  the directive identifying which table in the join is to be preserved. Compare the following queries:


The first, the LEFT OUTER JOIN (Figure 2), returns 104 rows (one for each row in the CUSTOMER table) and there are several NULL values because not every record in customer has a matching record in contacts.

However the second, the RIGHT OUTER JOIN (Figure 3), specifies that all records in the contacts table be included. Now we get only 90 records returned. This is because whilst every contact is related to a customer, some customers have more than one contact and others have none. 

The third, and much less common version of the outer join is the FULL OUTER join (Figure 4). This combines both the results that would be obtained by running first a left outer join, and then running a right outer join with the same set of criteria.

The result set from a full join actually contains three separate sub sets of data;

·         Records that match in both tables (i.e. the Inner Join set)

·         Records from Left table with no match (i.e. the Left Outer Join set)

·         Records from Right table with no match (i.e. the Right Outer Join set)

Cartesian Join

More properly this refers to a ‘Cartesian product’ that results from having either no join condition specified, or having a join condition that is inappropriate or invalid. What happens is that every record in one table is matched with every record in the other. In the case of the query below, the 90 customers and 90 contact records would produce a result set containing 8100 rows (90 x 90 = 8100).

In Visual FoxPro (prior to the introduction of the CAST() function in Version 9.0) there was no way to define calculated columns with a data type of either “Integer” or “Memo” as part of a query. The only practical way to achieve this was to define a dummy table with one empty row containing a column of each data type and then including the table with no join conditions, in the query. The resulting Cartesian product adds the necessary blank column(s) to every record in the result set. Thus to add an integer column and a memo field to a result set in VFP 8.0 you would need to do this:
However, it should be noted that while undoubtedly useful in certain specialized circumstances, the usual cause of a Cartesian product is a mistake in your SQL!

Union

A union combines the result sets generated by separate queries into a single result set. In order for this to work, the result sets from the included queries must generate the same number of columns, in the same order and of the same data type. (Note, some implementations, including SQL Server, will accept certain data types mismatches providing that they can be implicitly converted. However, it is generally considered good practice not to rely on such implicit conversions and to ensure that the data types are really compatible in the result sets).

There are two forms of the union statement, UNION and UNION ALL. The first returns all rows that appear that appear in either or both result sets, and discards duplicates. The second retains all duplicate records and effectively appends the second result set to the first.

The difference is subtle, but critical – especially when creating complex reports where data from various (unrelated) tables must be combined, or multiple sets of conditions apply. By using UNION ALL, you can create a single dataset from disparate data.

The queries in the example above return the total sales of condiments (pcat_pk = 2) by product alone, or in combination with the same data on confections (pcat_pk = 3). A total of 12 Condiment products, and 13 Confection products have been sold. Consequently the plain UNION returns the 25 unique rows (12 + 13) that qualify. The UNION ALL returns all 37 rows that qualify (i.e. 12 + 25).

 

That'll do for now, next time I'll take a look at sub-queries.

posted by andykr | 4 Comments
Filed Under:
More Posts Next page »