Announcing: Bag O' Regex

Bag O' Dreams

Build history

Installation note

Latest builds

but allows http://kitchen-sync.com/bag/BOR.html
• Files changed: bag.jar.

• 03-17-2007 New ~Regex report: finds all dreams not containing the user-entered regular expression.
• Added a command to the search term formatting drop-downs. (|). This one converts a space or comma delimited list of words into an alternating regular expression.
  Example "Rooms":

  attic bathroom basement "living room" "dining room" bedroom cellar closet den hall hallway kitchen porch room staircase stairway

  becomes

  \b(attic|bathroom|basement|living room|dining room|bedroom|cellar|closet|den|hall|hallway|kitchen|porch|room|staircase|stairway)\b

• The exclude links on cluster reports are now a little more "dynamic". If you click on one a new report is generated with any excluded dreams removed. The new report has all of the information of the parent report except for any matching terms.
• A draft of a new plugin profile report is now available: Self-reflection. This one helps you identify self-reflective awareness, metacognition and lucidity in a set of dreams. Just save the link as a text document in the plugins folder and restart Bag O' Dreams. The report will show up in the lower left drop-down on the Search panel.
  Helps answer questions such as;
  • Is the dreamer present in the dream?
  • If so, is the dreamer active?
  • Is the dreamer exercising choice or control?
  • Is the dreamer lucid?
  See references:
  • Kahan et al: Similarities and differences between dreaming and waking cognition: an exploratory study. Conscious Cogn. 1997; Mar;6(1):132-47.
  • Purcell et al: Dream self-reflectiveness as a learned cognitive skill. Sleep. 1986;9(3):423-37.
  • McNamara et al: Counterfactual Cognitive Operations in Dreams. Dreaming. 2002; Vol. 12 No. 3,
  • Rechtschaffen: The Single-Mindedness and Isolation of Dreams. Sleep. 1972; 1, 97-109.
• 03-10-2007 Added Expand and Lemmatize drop-downs to the two search term fields on the Co-ccurrence tab. (See the note for the build of 01-26-2007.)
• Added word totals to the Plugin Profile reports.

Previous builds

• 02-25-2007 Fixed a bug in the Plugin Profile reports: the indicator bars were twice as large for negative results as for positive.
• If you enter the letter 'g' in the search field of a result window before you run a Plugin Profile report then the result will use the log-likelihood score (G2) instead of the h-score. Log-likelihood is a little more conservative about the significance threshold of a feature when the cluster of dreams is small.
• 02-17-2007 Default text for new documents. When you click on the New dream button the new document is prefilled with the text stored in the settings/default.txt file. This includes a couple of flags that allow you to add the current date and the last used dream folder name as well as the cursor insertion point.
  • Use "%ds" for a date stamp.
  • Use "%dir" for the name of the directory where the dreams are stored.
  • Use the pipe character "|" to set the cursor position.
• The Index report on the AutoCluster panel has been removed. It wasn't really an index in any typical sense. Instead, the words in the term table are now hyperlinks, that look up all dreams containing them when clicked on.
• 02-10-2007 Changed the Terms at freq. report (Search panel, middle combo) to show all counts at once. (It no longer takes a value from the user for this.) Results are displayed in a table showing the number of terms which occur in only one dream, two dreams, three dreams, etc. The results are links which display the terms occurring at that document frequency.
• The end of the report now displays terms occurring at selected document frequencies; 1, 2, 3 to 4, 5 to 8, 9 to 16, etc. (This is only displayed when the dreamset contains at least 25 dreams.)
• Fixed the "Co-occurrence" command on the Search panel. It was incorrectly trying to run a plugin report.
• Changed the scoring for the Proximity report. It now gives some weight to the total number of matches.
• New feature: after adding a set of dreams you are automatically taken to the Search panel (unless you move to some other panel while the dreams are being added).
• 01-26-2007 New plug-in profile reports. These load from text files located in the new plugins folder. When Bag O' Dreams starts up it installs a profile-type report for each file it finds in the plugins folder. The name of the report is taken from the name of the file. These reports show up on the Search panel (leftmost combo) and in the Search menu of every result window. The plugin files must be in plain text format (".txt") and must containing lines in the same format as the old Quick Profile report. The lines are comma-separated lists of search terms, where the first term is the name of the category.

  Example file "Characters.txt" contains;

  Males ,he,him,his,boy,man,men,guy,fellow,male,...
  Females ,her,she,girl,woman,women,lady,female,...
  Family ,mom,dad,brother,sister,mother,father,...
  Human ,acquaintance,actor,addict,administration,...
  Animals ,animal,ape,bat,bear,beaver,bird,bull,...
  Strangers ,(two|pair of|twin|unknown|unfamiliar...
  Nationality ,\b(armenian|lebanese|hispanic|russian|american|...)s?\b?

  When you select the "Characters" item you get a report showing the percentage of each category and its significance (h-score).
  Note that regular expressions can be used in place of comma-separated lists (the last two in the example above). Regular expressions are now automatically detected and no longer have to be stored in a separate file.
• Added a few plugin files to get you started.
• The last line in a plugin file that starts with a # is treated as a comment and is included in the footer of the plug-in profile report. Use this as a description for the categories in the file.
• The old Quick profile report is now a plugin titled "QP". The Readability portion of this report has been split off as a separate report.
• The SMOG index (readability measure) was completely wrong. (There was an error in the way multi-syllable words were counted.) This has been fixed.
• New Proximity report shows dreams where at least two of the search terms were found within a window of 10 words (any order). If you want a different window width enter it in the threshold field. Only works if you enter at least two search terms.
• New Lemmatization of space and comma-seperated lists available on the Search panel (tiny icon combo next to the search terms field). This is a cheap and dirty lemmatization which removes words which have the same stem form the list. This is useful if you want to use a thesaurus look-up in a search and you have the stemmer turned on. Without it, you'll count the same stemmed term more than once. To use this function select the "L-" icon on the little drop-down combo box to the right of the Search terms field.
• Moved the Expand command to the little drop-down combo box next to the Search terms field. To expand your terms select "E+".
• Regular expressions searches are now case-sensitive by default. These include the Regex commands on the Search panel and the Search menus in dream and report windows. The regular expressions used in the plugins folder are also case-sensitive. If you want to make a regular expression case-insensitive prefix it with (?i).
• Improved the Names report for names that are also common words; Joy, Hope, Grace, Cliff, etc. The technique for determining significant names uses a threshold of 1 per 10000 words to determine "important" characters but it used to count all lowercase version of "names" and even matched based on shared stems so that 'joyful' was counted as a match for 'Joy'! Fixed.
• 01-01-2007 Cleanup of Search panel: it now has four drop-down combo boxes instead of two.
• Added a new combo for "profile-ish" reports. Reports that tell you something about the dreamer or the set of dreams and don't use search terms.
• Added a combo for the modules folder and "utility" methods such as Split and Concatenate.
• Moved all of the reports that use search terms to a single combo. These are the most commonly used.
• Moved all of the remaining reports to their own combo. These are reports that don't use search terms.
• Completely removed the following seldom used reports;
  • Neighbors
  • Pairs
  • Subsumption
  • Cluster (cs)
• The "matches only" version of the Regex reports can now be searched within: the results window's search menu works.
• Minor change to the Profile report: now finds "our son|daughter|children" in addition to "my son|daughter|children".
• 12-25-2006 Changes to Color Profile;
  • Now reads in the hex color value from the colors.txt file. This allows you to rearrange the colors and add new ones. The format is:

    Color name,hex value,regular expression

  • Added "Pink" Pink,#ff99c2,\bpink(er|ish)?\b because, in the west at least, it is considered a basic color category.
• Added alternate flavor to the Regex reports: if you enter "-1" into the threshold field of the Search panel the regex results will show just the matching expressions with counts.
• 12-22-2006 New Color profile report is designed to look at 10 color categories;
  • Useful to anyone trying to test the significance of color in dreams.
  • Shows color words as a percent of all words in a dream or cluster.
  • Colored bars indicate a color's significance by h-score.
  • Also displays the relative frequency of each color.
  
  Controlled by a list of regular expressions: "settings/colors.txt". Edit these only if you know what you're doing! Do not change the order of the colors. Do not add new lines.
• 12-17-2006 The Quick Profile report now uses a second file for user-entered reqular expressions: "settings/regex_categories.txt". Each line in the file must contain a name for the category, a comma and a single regular expression.

  Bigwords,\b[a-z]{7,}\b

  The number of matches is reported as a percent of words in the cluster of dream set.
• 12-06-2006 The Quick Profile report now uses the h-statistic to measure the difference from the norm in each category. The length of the color bars is directly proportional to the h-score. Categories that are significantly different from the norm are now marked to show their p level.
• 11-25-2006 The Profile report has a new category called Era. It shows the era of a dream set in 25 year chunks from 1850 to the present. A dream set from before 1950 will have zero matches from 1950 - 2000. This report works by looking up inventions, famous people and things that originated in each of the six eras covered. (There are very few dream sets from pre-1850 so I didn't bother covering them.)
• The Era table also includes a list of "Old fashioned" words: mostly consisting of terms that have decreased usage since about the 1960s.
• 11-20-2006 The Profile report has been reworked with two new categories of results added;
• "Affiliation and otherness" - gives a summary of characters in 4 sub-categories that could reveal a little about the dreamer's attitude towards others.
  1. Race references: What do references to people of same/other racial and ethnic groups reveal about the dreamer?
  2. Sexualities: What do references to people of same/other sexual preference reveal about the dreamer?
  3. Religion: What is the dreamer's experience of religious groups?
  4. Politics: Can you tell the political leaning of the dreamer?
• "Personal emotions and fortune" - gives a summary of emotion "close to the dreamer" in 4 sub-categories;
  1. Positive emotions: "I'm so happy", "I think she is a wonderful", etc.
  2. Negative emotions: "I am suspicious", "I burst into tears"
  3. Good fortune: "love me", "good place for me", "helps me"
  4. Misfortune: "complain that no one is helping me", "shoots me", "attack me"
  All of the results are displayed with their frequency per 10000 words to allow comparision between different dreamers.
• Both the Profile report and the Gender report now include links (in category names) that show you the content that Bag O' Dreams used to make its guesses about the dreamer's characteristics. Click on the "Religion" link and you'll get a report listing most occurrences of words directly related to religion; "jew, christian, worship, hindu, etc."
• Fixed a bug in the AutoCluster reports: the summary Characteristic terms link was broken.
• 10-29-2006 Gender report: added a new test of the dreamer's gender based on the Gender Genie page (http://www.bookblog.net/gender/genie.html).
• 10-28-2006 New Cloud 9 report shows words in a dream arranged by nine categories;
  • emotions, horrified, sorry, happy, ecstatic, etc.
  • charcaters (and animals): he, she, fireman, snake,
  • body: sex, eating, sensation, clothes,
  • setting: highschool, church, bedroom,
  • objects: box, stove, walnut, car, etc.
  • actions: ran, opened, said, feel, throw,
  • thinking: thought, knew, remember, decided,
  • visual: red, yellowish, appear, sight, dark, translucent,
  • space/time (size, extension): now, soon, big, ahead, minute.
  The size of the words is proportional to their frequency in the dream.
  Links from report display (tag) each category's terms in context.
  Word lists for the report are stored in settings/cloudnine.txt.
  Cloud nine is found in the Search menu of individual dreams.
• 10-17-2006 Added more "exclude" links to most (all?) reports that needed them.
• Added a "Global search" link to the global version of the Term Table report so you can enter the whole term table into the search field.
• The Wrap command no longer wraps lines that begin with #. This makes it easier to reformat dreams downloaded from DreamBank. In particular, the title line of these dreams would wrap with the first paragraph of text, effectively commenting out the text.
• Fixed a bug which happened when you removed a dream text from the working dream set and than ran any report that uses regular expressions.
• 10-13-2006 New "exclude" links in cluster reports. If you click on an exclude link, you'll be given the option of removing the dream from subsequent cluster searches. Useful if a dream doesn't really belong in the cluster result. Example: you search for flying dreams and the cluster includes a dream because it mentions a fly (insect). You can clear all excluded dreams by selecting the "Clear excluded" item from the second drop-down on the Search panel.
• You can now view the HTML source of any report by selecting the "View source" item in the File menu of the report's window. Useful if you want to copy part of the result and paste it into another HTML document. Useful to me for debugging formatting issues!
• Minor clean-up of the menus.txt and categories.txt files in the settings folder.
• 09-29-2006 The Expand report now uses a different test of term association to pick words that should be included in the expansion of the query. The test gives less weigh to higher frequency terms. The reason for excluding high frequency terms is that, as synonyms, they can't really be helpful in finding new dreams with the same content as the query if they occur in most of the dreams in a set. They may increase recall, but they definitely decrease precision.
  By the previous method the term "fly" expands to; ground fly float ability land air inch in one dreamer's dreams.
  The new version gives; ability fly float land air branch.
  The term gound, while occurring in 2/3 of flying dreams, also occurs in twice as many non-flying dreams.
  Excluding it from the results increases precision (when used as search terms in a log-likelihood clustering): 35/43 flying dreams using the old terms versus 36/37 for the new terms.
  The Expand report that is run from a cluster of dreams uses the old test of asociation, because, when the number of dreams is much less than the size of the whole set of dreams the new method returns very few results.
• 09-27-2006 Changes to Frequency report: when you used a threshold in the Frequency report it used to show only terms that were greater than the threshold. It now shows all terms, but indicates higher than threshold terms in red.
• Cloud report display is now a little roomier. The space between terms was always at the base font size, which left "large" words crammed together. Fixed.
• 09-22-2006 Added "tag matches" links to the three "Cluster (xx)" reports, "Cluster by date", "Related", "Dreams with", "TIC" and the "Left w/o Right" reports. When you click on these links the result terms are tagged (color highlighted) in the dream where they were found.
• 09-19-2006 The Expand report now uses "+/-" prefix to include/exclude results from the expansion query. If you do an expansion with these search terms "water -fish" you will get terms related to water but not to fish. A "+" prefix forces the inclusion of terms that would be excluded due to lack of significance. Both plus and minus terms are "sticky": once you prefix a term with "+" or "-" it is automatically re-entered in the search terms field after expansion.
• 09-17-2006 The Expand report has been improved. The results now only include terms which were neighbors of the original search query terms. This prevents a kind of "chain matching" of secondary terms. The Expand report is also now a little more verbose in small sets of dreams.
• The Terms at freq. report now takes a "hint" from the search terms field to show terms greater than or less than threshold frequency. Use "lt" for less than, "gt" for greater than. If you enter "gt" in the search field and "9" in the threshold filed the report will display all words that occur in > 9 dreams. (Leave the search field blank and you get the default behavior: in this case all all word at a frequency of 9.)
• K-means now displays a "history" of the q-score at each iteration.
• Fixed a bug in the TFIDF report, which sometimes caused all vector elements to be set to "0.0".
• The Frequency report now lists all non-matching terms in a list at the bottom of the report rather than in the table. This makes it easier to find the matches in long lists of words.
• Window titles include the dream's file name in reports run from individual dreams.
• The Settings panel's "Max doc occurrence" is now the same, regardless of the use of the Porter stemmer. (With AutoAdd checked.) I found that it didn't make much difference when this setting was changed.
• The "settings footer" shown at the end of many reports now includes the "Number of dreams to add" that was selected by the user.
• Quick Profile report now uses relative difference score to control the size of the red and blue bars that indicate difference from the average.
• Some of the "Tagging reports" now handle terms that were previously skipped because they messed up the html. These words include "color", "size", "background" and "style".
• The Regex report and the Find and Tag report now format their paragraphs a little more consistently.
• 08-26-2006: The Cloud report now shows words unique to a dream or cluster in a purple font color. Previously they were shown in red.
• Made several more adjustments to the matching method used by the Cloud, Redact and Tag reports. This method matches a search term like "make" to most of its common suffixed forms; "makings", "makes".
• Increased the "Max doc occurrence" setting used when you check AutoAdd on the Settings panel. What this does is let a few more of the most common terms into the bag of words model.
• "Detached" cluster windows from the AutoCluster reports now list the cluster's characteristic terms in their title bars.
• 08-17-2006: The Quick Profile report now indicates difference from average value by colored bars like this:

  category	low	high	cluster	all dreams
  Neg. emotions			0.65	0.69
  Pos. emotions			1.03	0.68
  Self reference			8.22	8.73
  Social			5.55	6.41
  Dialog			0.33	0.82
  Pronouns (not I)			3.92	4.7
  Males			2.61	1.78
  Females			0.82	1.54

• The Cloud report now displays words that are "notable by their absence". These terms are show in blue. "Positive" terms are now shown in red like this:

  light old pick rest taken good second hand tried floor turn girl try little people look say sternly obeys flashed

  Entering my room, I try to turn on the overhead light but it flashed on for only a second and then went off. I tried it again and then another light and then my old table light. No good, I must tell them. I go out and look for the books I had laid on a table outside. Ask several people about them and see several piles of old books but not mine. Who could have taken them? I see a little girl throw something on the floor. I go to her, rest my hand on her shoulder and sternly say, " Pick it up." Repeat 3 or 4 times until she does. "Now throw it in here." She reluctantly obeys.

• The matching/tagging method is slightly better.
• 08-14-2006: The Cloud report can now be run from clusters of dreams. It works the same as the single-dream version except that only the cloud terms are shown.
• 08-13-2006: The Cloud report now includes terms that are unique to the dream.
• Adjusted the code that matches a word's root with suffixed forms. This improves the Cloud report, the Find and Tag search and Dictionary panel lookup.
• 08-12-2006: New Cloud report shows most significant terms of a dream in context. The log-likelihood statistic is used to measure how significant a term is. The more significant the term, the larger the font size.
  Run the Cloud report from any dream's Search menu.
• 07-30-2006: Added new lexical measures to QuickProfile report;
  • Total Number of words,
  • Average number of words (per dream),
  • Vocabulary: number of distinct words,
  • Hapax legomena: number of words occurring in only one dream,
  • Hapax dislegomena: number of words occurring in only two dreams,
  • Average and total sentence length,
    Note that finding sentence borders is not a trivial problem. The sentence "Did Mr. J. drive... to N.C. in 12.8 hours!?" would be counted a ten sentences if you simply treated [.!?] as sentence borders. My regular expression doesn't handle "Mr." - and probably some other cases - so it's an approximate measure.
  • Average and total number of syallables.
    Again, not a trivial problem. I use a regular expression that seems to be about 90% accurate.
  • Gunning-Fog index: an approximation to a widely used measure of readability. I use 0.4 * (average sentence length + % words with 3 or more syllables). A score of 6 for easy to read to >16 for very difficult to read text.
  • A few other readability measures.
  • Lexical density is now displayed as a percent.
• Redact report now uses colored rectangles for redacted words. The colors are keyed to the category of word removed.
• 07-17-2006: New Redact report: replaces most content words in a dream with gray squares:             . The words removed are the same ones that would be color-highlighted by an All Tags report. This allows you to check your word lists to see what type of content isn't being tagged. Once you remove all known content words it's easier to see the one's you missed. This might also be an interesting test of if-it-were-my-dream interpretation idea: remove all names, characers and emotion words from an unfamiliar dream and then replace them with ones you know to make it your dream. Run this report from the Search menu of any dream.
• 07-11-2006: Forced a global refresh of the Quick Profile results when switching between sets of dreams.
• Minor update to the categories file.
• 07-03-2006: Quick Profile results are now searchable. The category links now place their word lists in the search field of their own window. This is useful if you have QP results for a cluster of dreams and want to see exactly which words matched in each category.
• 06-27-2006: The Quick Profile report now shows strong results in bold so you can pick them out at a glance. (Still no test of statistical significance though.)
• Also made the report a little faster: "global" results aren't updated unless the QP is run from the Search panel.
• 06-24-2006: New Dreams by size report available on the Search panel. This allows you to show all dreams with a word count between the min and max you've selected in the Date and Size Limits dialog. This report ignores the date range limits unless you've actually enabled them.
• Fixed a bug which happens when you select Date and Size Limits that exclude all dreams from the set. This would crash a few of the reports.
• Fixed a bug that prevented you from running more than one report at a time from a cluster result window.
• 06-20-2006: The Quick Profile report is now customizable. Add new categories to the file: settings/categories.txt. Each category is a comma-separated list of words and phrases. The first word in the list is the name of the category.

  Dialog,answers,asks,reply,said,says,speak,talk,tells,told

• Note that a few categories are hard-coded (Strangers, Negations, Punctuation, Big words, Lexical density) because they use regular expressions and/or non-percent word calculations.
• The category names are now links which, when clicked on, put the word list or regular expression into the Search panel's text field.
• The Hall/Van de Castle norms are no longer displayed in the report. Instead, the results for all of the dreams in a set are compared to individual dreams and clusters. I've set up a link showing the norms for the current set of categories: male norms, female norms.
• 06-06-2006: Quick Profile report: added a few new categories;
  • Inclusive: we, our, together, with...
  • Exclusive: them, other...
  • Passive: watch, wait, sit...
  • Informal: huh, uh-uh, oh, duh...
  • Discrepancy: could, should, ought...
  • and made a few adjustments to other word categories.
• Results that are lower that the norms are now highlighted in blue: high results are red. Usually a difference of > 33% from the mean of the male and female norms is used as a threshold. In a few cases the threshold is higher or lower because these categories seem to vary a lot or very little.
• Profile report (not Quick Profile) no longer has a verbal info section, since this is now handled in the Quick Profile.
• Gender report gave incorrect counts for gender specific body self-reference. Fixed.
• Frequency report now shows a line labeled "% of words": 100 * (Total count/total number of words). This value is what the Quick profile uses (in most cases) so you can it for custom word lists.
• 05-23-2009: Change to Profile report: the values for the male and female norms display different results, depending if the stemmer is on or off.
• Also added Lexical density measure: number of distinct words / total number of words.
• 05-21-2009: "Globalized" the Quick Profile report. You can now get a quick profile on any cluster or the whole set of dreams.
• Added a several new categories to the report. See link above for details.
• Changed the Relative Difference report so that it also displays dreams with zero scores.
• 05-10-2009: Changes to Quick Profile report; Added more words to most categories. Added "Big words" = % of words > 6 letters long. Six of the word lists are now read from a file "settings/categories.txt" so you can edit them if you like. Just add words or phrases to the line labeled with each category. Each entry must be separated by a comma. (Doing this will throw off the values for the norms. Send suggested words to me and I'll fold them into the next build.)
  Note that this report (currently) depends on using the Porter stemmer. A future release may be more flexible so that "thought" would match thoughts, thoughtful and thoughtfully.
• 05-06-2006: New Quick Profile report: gives you a very quick idea of a dream's content for a dozen word categories; positive and negative emotion words, sexuality, self-reference, cognitive words, motion, and more. It also displays results for the Hall/Van de Castle norms for comparison. All word categories are reported as a percent of the total number of words in the dream. This report is only available on the Search menu of individual dreams.
• Change to Random Sample: previously, if you asked for 100 random terms (or dreams) you could get the same term (or dream) more than once in the results. Now you are guaranteed to get 100 unique random terms (or dreams) unless your set of dreams is too small.
• Fix for certain reports that give summary information about a set of dreams. Previously, if you had Date/Size limits turned on so that ALL dreams were filtered out and ran a summary report, it would give either a blank report or an exception. Now these reports do nothing if all dreams are filtered out of the set
• 04-22-2006: New TFIDF report on Search tab. TFIDF stands for "term frequency - inverse document frequency". This is a specialized search that takes a list of words and creates a ARFF format file where each line is a (sparse) vector of TFDIFs for each dream, normalized to unit length. The search terms are kept in a text file (settings/tfidf.txt). The file must contain a comma-separated list of words on a single line. If you want to use the report with WEKA then you'll need to avoid most non-alphanumeric characters in the tfidf.txt file.
• Minor fix of the way Bag O' Dreams handles some situations where a file or folder is missing. Example: the modules folder is for optional modules but Bag O' Dreams wouldn't start unless it was present.
• 03-25-2006: You can now show a list of all topics in the Dictionary panel. Select the first item on the tiny drop-down selector to see all topics. Select a topic and click the Find button to see its definition. This only works for the Dictionary - not when the Thesaurus or Phonic lookup are selected.
• Fixed a bug in the "number of dreams to add" selector on the Settings tab. It got out of sync because of the new option to add 25 dreams.
• 03-21-2006: New Relative Difference report on the Co-occurrence panel. This one takes a simple count of the right and left terms and reports the difference as a percent. Useful for finding past versus present tense dreams, dreams with more male characters than female, etc.

  Left terms: is be am are have say
  Right terms: was had were said
  

  The report scores dreams using past tense as -100 to -1, present tense as 1 to 100. Use the threshold field to distinguish cases like these. "-1" gives only dreams scoring less than -1, "1" gives dreams scoring greater than 1. (Enter "0" to show all.) Results are color-coded to match the search terms.
• Added ARFF report to individual dream documents.
• 03-15-2006: Fixed a bug in the ARFF report. Also added new attributes. Details in the Update section of the ARFF Report page.
• 03-11-2006: New ARFF report: generates an file in arff format that can be used by the Weka classification package.
• 01-28-2006: Added a line to the Profile report that shows the ratio of past to present tense usage in dream descriptions.
• Renamed the "Cluster/Frequency" report to "Cluster by date" report.
• Modes report now shows a red progress bar during the first-time-only calculation of normative values.
• 01-11-2006: Added a threshold to the Names report. To find the most significant people to a dreamer run the Names report with a threshold of 1.0.
• The Expand report no longer displays a result window. Instead the expanded terms are entered directly into the text field from which the report was run.
• Got rid of the "+/-" toggle button on the Search Panel because I seldom used it.
• Replaced it with a Save terms button which saves the current terms for the current menu item.
• Removed Save terms command from the second drop down list on the Search panel.
• Got rid of the Mysterium Coniunctionus report because I never use it.
• Replaced the three "Random" reports (Random dream, Random cluster and Random Terms) with a new "Random sample" item.
  • To get a random dream just chooses Random sample with the threshold field set at 0.
  • To get a random cluster of dreams enter a threshold value greater than 1.
  • To get random terms, enter the letter 't' in the search field before selecting Random sample.
• 01-02-2006: changed the names of the Cluster reports and added a new one;
  • "Cluster (tw)" - the original term weighted cluster algorithm.
  • "Cluster (ll)" - cluster based on log-likelihood statistic (formerly named "Cluster by modes").
  • "Cluster (cs)" - new cluster search based on cosine similarity.
• 01-01-2006: Made the Expand and Pairs reports about three times faster.
• Removed MIM and Jaccard measures from the Pairs report.
• 12-23-2005: Added Frequency and TIC (terms in context) commands to the Search menu of individual dreams.
• Rearranged the Search menu commands to be a little more consistent.
• Fixed a bug in external links, which occurred with some versions of Internet Explorer.
• 12-14-2005: Fixed a bug in Date and Size limits panel. The layout was broken in the previous build.
• Removed the "Stop More" panel because I never use it.
• Changed the Related report so that it uses a different term weighting formula. I think this improves the results slightly.
• 11-26-2005: Fixed a bug in the dream Term Table report. The number of words reported for a dream was always zero.
• Changed the way reports are positioned. Result windows are now displayed at the point where the little progress indicator window was last positioned. Reports that take a long time to generate can be repositioned by dragging their progress dialog to wherever you like.
• Changed and renamed the First/last words report. It now shows middle words too. The number of words shown in each section is whatever you entered in the threshold field (but if you enter 0 or less it shows 10 words). The format of the report is now plain text. Each section (first/middle/last) starts with a document marker so you can split the result into individual dream documents. (Save the results as a text document and then use the Split command.) The documents also contain one of three "tags" _FIRST, _MIDDLE or _LAST. I did this so you can do a cluster search for the first or middle or last words to see if they contain significantly different features from other sections of dreams.
• 11-13-2005: New Index report on AutoCluster panel. After you run at least one AutoCluster report (k-mean, fuzzy-k or r-cluster) an "index" is generated. The index is the intersection of each cluster's characteristic terms and the top scoring dream (spotlight dream) of the cluster. The report shows the list of spotlight dreams with their index terms. The idea is to make the characteristic terms more concrete by filtering them through an actual dream document. As you run more AutoCluster reports more terms and dreams are added.
• The Frequency report now uses the threshold field. The threshold value is used as a lower limit for the frequency per 10000 words of each term. Try the following;
  1. Run a Names report.
  2. Enter the results in the Search panel.
  3. Run a Frequency report with a threshold of 1.0.
     The results show the most important people in the dreamer's life.
• The Most Significant Terms report has been changed. It now uses a lower level of significance when run from individual dreams. The standard that is usually recommended for the statistical test limits it to terms where the observed or expected frequency is at least 5. This is fine for clusters of dreams, but usually shows no results for single dream documents. Lowering this level to 2 returns more results - though the results that aren't strictly significant. Since the goal is to reveal the dream's content, a result which usually gives no information is of little use.
• 10-16-05: New Bigrams report on the Co-occurrence panel. This report takes the terms entered left and right text fields as a two word phrase and finds it in the set of dreams. The Width setting has no effect on the results - terms must always be pairs, separated by no other words. The results are sorted by a log-likelihood measure, which shows how likely they are to be paired. (A score of over 3 ought to be significant.)
• Split the Gender report off from the Profile report and made it more verbose. It predicts the sex of the dreamer.
• The "Most closely related" section of the Interpretation report now list the six best terms found in the related dreams.
• Changed default setting in the date/limits box from "50 to 500" to "5 to 5000". (I look at some fairly large dream reports.)
• Blocked the running of some reports from individual dreams when the selected limits would exclude that document from the set.
• 09-27-05: Removed k-sharp as a separate item on the AutoCluster panel. Added sharpening to k-means and k-fuzzy searches. "Sharpening" is the removal of dreams from clusters when their score is less than the user set threshold. If you don't want sharpening, set the threshold to 0.
• Made the results of the Names report a search link.
• 09-21-05: Bag O' Dreams is now free for individual use. See the License agreement on the About Panel for details.
• 09-19-05: Removed Correlation report from Co-occurrence panel because I decided that it was bogus. (A better test of correlation is a cluster search for one theme followed by a Term Significance report for a second theme, run from the found cluster.)
• Made the Anti-Cluster report explicit. It can now be run by selecting "~Cluster" from the Search panel. It returns all dreams that score less than or equal to the user entered threshold (usually 0) for the terms entered in the search field. Finds all dreams without a theme or feature.
• 09-13-05: New "Left - Right" report on Co-ocurrence panel. This is similar to the "Left w/o Right" report. It uses the log-likelihood score to match each of the Left search terms in each dream. It then subtracts from a dream's score if it contain Right terms in any significant number. A threshold of 4 to 6 seems to give the best results.
• New "Left w/o Right" link at the bottom of the Term Significance and Most Significant Terms reports. This one enters terms with a significance level of "++" or "+++" or higher into the Left search field of the Co-occurrence panel. Terms at "--" or "---" get entered into the Right field. This lets you do "Right w/o Left" and "Right - Left" searches more easily.
• 08-15-05: Phonic lookup of single words is now available on the Dictionary tab. If you enter a nonsense word like "plustord" in the Dictionary's text field and select "Phonic" you'll get a list of somewhat similar-sounding words like; "palaestra, palisade, palsied, pellucid, pilaster, placed, placid, placidly, plaster, plastered, plastic person, plastic surgery, plastic, plasticity, pleased, polestar, pollster, pull strings, pulsate, pulsating", You must have the thesaurus files installed in your setting folder for this to show up.
• The Names report is now a little smarter about what counts as a person's name.
• 07-13-05: new version of k-means called "k-sharp", removes dreams from clusters when their closeness to all clusters is lower than a user set threshold. The threshold is entered through the "r-threshold" field on the AutoCluster tab. Removing outliers from cluster might sharpen the results in terms of q-score and thematic content.
• The "Fuzzy k-means" report has been renamed "k-fuzzy".
• The Profile report now contains information about the dreamer's vocabulary.
• New "Words at length" report displays all words at a user entered length. (Found on Search tab.)
• New "Words lengths" report displays the number of words found by length (1 - 24). (Found on Search tab.)
• Added mutual information measure "MIM" to Pairs report. Higher MIM means that two terms are more likely to be interchangeable.
• New Regex menu just for regular expressions. When you select it a Regex report is run with the regular expression you entered.
• The "Emotions" menu has been renamed "Modes" since it includes many (new) items that aren't emotions so much as cognitive modalities,
• Updated the MenuEditor so that it handles regular expressions better.
• 04-09-05: Changed the default minimum doc occurrence setting used when AutoAdd is selected. Setting it higher reduces the size of the term table by a factor of 10. This makes some searches considerably faster without significantly changing the results. The k-means, runs about 3 times faster than before.
• 04-02-05: The Profile report is now separate from the Dreams info report.
• Reorganized the search menus between plain text documents and results windows so they are more alike.
• Deactivated most searches when they are irrelevant - in reports that don't contain dreams in their results.
• Switched the behavior of term links in cluster reports from global to local behavior. If you click on a "Best Terms" link, the terms get put in the report's text field instead of the Search panel's text field.
• "Search within" links have been replaced by "Global search" links. Click on one of these and the search terms are entered in the Search panel's text field.
• Added the Jaccard coefficient to the Pairs report.
• 03-13-05: New "Most Significant Terms" report: works like the Term significance report, but uses the dream set's term table instead of terms entered by the user.
• 02-12-05: The "Dreams info report" now includes a Profile section that makes predictions about the dreamer's gender, family, marital status and employment.
• 02-10-05: Added new column to the Frequency report: "per 10K". This shows the expeced frequency of the term per 10,000 words.
• Added a gender prediction to the Dreams info report. This uses the frequencies of certain words (a, the, an, she, to, of, color_words and quantity_words) as predictors. (Based on the paper: M.Koppel, S. Argamon and A. Shimoni (2003), "Automatically categorizing written texts by author gender", Literary and Linguistic Computing 17(4), November 2002.)
• 02-06-05: New "Cluster by modes" report. This one is a cluster search which uses the statistical significance of the search terms in a dreams as a test for inclusion in a cluster. This is the same test as is used in the Term significance report.
• 01-29-05: Added Modes and Term significance reports to individual dream documents - under their Search menus.
• Fixed bug in the way report headers and footers were displayed: only shows up if you're using JRE 1.5.
• 01-24-05: New Terms significance report. This one compares the frequency of search terms in a cluster with the expected frequency (based on the whole set of dreams and adjusted for the average length of the cluster's dreams). Terms are scored and marked with +/- signs if they are significantly over or under expressed within a cluster.
• The Modes report now includes weighting for the average number of words in the cluster's dreams.
• 01-15-05: Added new significance level indicators to the Modes report. Also only show most significant modes.
• Pairs report now has significance indicators too. Now it only shows significant pairings.
• Expand report now relies on these significance indicators instead of the threshold field. The threshold now sets the width for the neighbors part of the expand report.
• Names report now uses a threshold to exclude very rare names.
• 12-26-04: New Modes report for clustered results. This report does a simple count of all items from every search menu (except for the User menu). The count is them normalized by dividing it by the number of dreams in the cluster. Results are compared to the expected percentage of each mode in the complete set of dreams.
• The cluster version of the Interpretation report now only shows items from the Typical and Personal menus.
• 11-21-04: Merged the Thesaurus into the Dictionary panel. Now you can look up items in either the dictionary or the thesaurus files from the same place. (You must have the moby thesaurus files installed for this to work.)
• I've split the k-means into two reports; "classic" k-means (the way it was before the build of 9-13-04) and the new fuzzy k-means. Fuzzy k-means merges the secondary cluster's dreams and characteristic terms into the primary clusters. (See the build notes for 9-13-04 if you don't know what secondary dreams are.) The idea behind fuzzy k-means is that one dream can belong to more than one cluster. A dream with a lot of driving and a lot of swimming will now show up in both clusters. This makes clusters and their characteristic terms a little larger.
• To support the addition of another AutoCluster report the interface now has a drop-down selector and "Find" button which works just like the one on the Search panel.
• Added links for summary characteristic terms and spotlight dreams to the r-Cluster report.
• New "Unique terms" report on the Search menu of each dream. This one displays terms that only occur in the selected dream.
• New "Unique terms within" report on the Search menu of result windows. This one displays terms that only occur in the report's dreams.
• New "Terms at frequency" report on Search panel finds all terms that occur at the entered document frequency. (Use the threshold field to enter the frequency.)
• The Regex (regular expression) search now shows the matches found directly on the report.
• The Correlation report on the Co-occurrence panel now uses top and bottom 5% and 1% of scores to determine significance. It no longer assumes that the distribution of scores is a normal one.
• 09-22-04: New "Characteristic terms" link in the Summary section of the AutoCluster report. This one extracts all of the characteristic terms links from the report and puts them in a new window.
• 09-19-04: Revised the measure for statistical significance of clusters again. Now it's based on the top 1% of scores for a cluster of a given size. This is a little more stringent, especially for small clusters. I did this because the distribution of scores in smaller clusters isn't "normal".
• Also fixed a couple of bugs that occurred when dreams had no text at all.
• 09-13-04: New feature: "Secondary dreams" Dreams often have more than one theme. The same dream might contain both "flying" and "driving". Since the k-means (AutoCluster report) can only assign each dream to a single cluster it might be missing themes. This is especially likely in dreams where two themes score very high, but one theme is slightly stronger than the other. One way to recover some of these lost themes is to create a secondary clustering of the dreams. In the secondary clustering the second best theme in each dream is found and the dream is then assigned to the cluster were this theme occurs. The practical result of all of this is a list of dreams that score second-best in each theme.
• Cluster reports now include links that detach each cluster (primary: 1° and secondary: 2°) of a theme.
• You can also merge the primary and secondary dreams together for combined searches.
• The Interpretation link, now comes in two flavors as well.
• New feature: "Spotlight dreams" in the AutoCluster report. This is a list of the first (top-scoring) dream in each cluster. Reading just these dreams ought to give you a feeling for the main themes in a set of dreams. "Spotlight dreams" appears as a link at the bottom of the AutoCluster report.
• 09-07-04: Fixed a small bug in the dictionary: if you selected text in a definition and then entered a new word to look up, the dictionary would look up the selected text.
• Also fixed this in the stand-alone thesaurus application.
• 09-06-04: Fixed a bug that occurred if you only had a single dream in your set.
• Fixed a bug that I introduced into the stemmer: error if you had the word "'re" [single quote followed by "re"] in your documents.
• "Better terms" have been renamed "Characteristic terms".
• The significance level for q-scores in cluster results has been raised from 95% to 99%.
• 08-15-04: Improved Better terms. Terms that only occur in one dream in a cluster are now excluded (because terms that occur in only one dream of a cluster can't be characteristic of the cluster). Better terms are now sorted by their document frequency within the cluster. Both changes make better terms a little more relevant.
• 08-05-04: New Export command on report windows' File menus. This is useful for cluster results. It concatenates all of the dreams listed in a report and saves them to a plain text file.
• 07-31-04: New option allows you to force words to be included in the term table even if they occur so frequently that they would normally be excluded. The normal settings used during the adding of a set of dreams excludes words which occur in more than 15% of dreams. Typically, one dreams of one's spouse or "significant other" so often that their name will be excluded and won't show up in automatic cluster searches. (I also noticed that I have so many flying dreams that the word "fly" occurs too often to be included.) The solution to this problem is to make an exception to the maximum doc occurrence cut-off. Add any words you'd like included (no matter what) into the new "anti_stop_words.txt" file in the settings folder. Add one word per line.
• Changed the text of the links that "detach" a cluster (in autocluster results) from "search within" to "detach this cluster".
• Changed the format of document links in saved HTML files so they're a little more compatible with non-Microsoft browsers.
• 07-28-04: Two useful perl scripts for download. One anonymizes a set of dreams - replaces known names with ones you supply in a list. The other script puts spaces around double hyphens so that Bag O' Dreams won't mistakenly classify "apple--" and "apple" as two distinct terms. Also included is a script that generates fake dreams with fake dream interpretations.
  Requires that perl be installed.
• 07-16-04: Fixed the way expressions with numerals are handled. Previously they were not included in the term table: "19th" was changed to "th". Now they are treated as letters.
• 07-15-04: Fixed the way hyphenated expressions are handled. Previously they were treated as phrases: "son-in-law" was changed to "son in law" for searches. Now the hyphens are left in.
• 07-12-04: Jungian tags. Here's a set of word list for Jungians (in zip format). See the "Read Me" file for details.
• 06-28-04: Freudian tags. These are word lists for use with the "All tags" report. They list words and phrases that are commonly believed (by Freudians) to symbolize latent (disguised) sexual content in dreams. Unzip the files and replace the ones in your "mytags" folder. See the document "Freudian tags.txt" in the zip file for more instructions.
• 06-16-04: The Co-occurrence and AutoCluster panel are now inaccessible to unregistered users. Please register your copy of Bag O' Dreams. Your support is greatly appreciated. Thanks.
• 05-19-04: Minor changes to the Pairs and Expand reports: they now favor rare terms a little more than common ones. Ideas mostly inspired by Reginald Ferber's Using Co-occurrence Data for Query Expansion: Wrong Paradigm or Wrong Formulas? (http://information-retrieval.de/ferber/homepage/pdf-ps/using-cooc.pdf).
• 05-07-04: Made the "Set operations" panel a stand-alone module.
• Small fix to AutoAdd: it now saves settings before adding the dreams.
• 04-27-04: Added mean number of words to the "Dreams info" report, for the year, month and day of week items.
• 04-25-04: Some minor fixes to the format of the AutoCluster panel reports.
• 04-12-04: New "Left w/o Right" report on the Co-occurrence panel. This report does a cluster search using the Left terms (words and phrases entered in the "Left" field). It lowers the score of any dream where a left term is close to a right term. This is another method that might help disambiguate themes which share descriptive terms (like flying and swimming). You could enter "fly float air" as left terms and "airplane fish swim ship" as right terms to exclude cases like "flying airplanes", "floating in a swimming pool" when you're searching for flying dreams.
• New section on recall and precision in the About panel list of documents.
• Report windows are now positioned above the window from which the search was preformed. This makes it a little easier to identify the source of the report.
• 04-03-04: Small change to AutoAdd: It now prompts you before adding an existing dream set. Previously, it just went ahead and added the set if it was checked.
• 04-01-04: One year anniversary! 86th release!
• Saving a dream after editing it now updates it in the current dream set (if it's already in the set) or prompts you to add it to the set (if it's not). Got rid of the "Add" button on dream document windows since it's functionality is all tied into the Save command now. (See build notes of 03-21-04.)
• Most reports now display the Settings used.
• Cluster reports now display the "probe" size just below q-score.
• 03-27-04: The AutoAdd feature (checkbox on Settings panel) now automatically calculates and sets the min and max doc occurrence settings for the set of dreams being added. This is based on the number of dreams found and whether or not "Use Porter Stemming" is checked.
• I'm evaluating a new approach to calculating expected q-score (a measure of cluster term coherence). This measure is currently obtained by calculating q for a large number of random samplings of clusters of dreams of different sizes. The new method is based on the q-score of clusters found by using sets of random search terms of various sizes. This overcomes the effect which the search terms (even randomly chosen ones) have on q-score: because the clusters are at least guaranteed to contain the search terms and this boosts the expected score above those obtained by selecting the dreams randomly. This makes the new measure more stringent than the older one.

  To access this feature go to the AutoCluster panel and click on the "Stats" button while holding down the control (CTRL) key. You'll see a progress window that says "Expected q by probe size". (Probe size refers to the number of words and phrases used in the search string - not the size of the cluster.) Use the resulting table to determine if a particular cluster of dreams is significant: look up the expected q-score based on the number of search terms used to find the cluster. Then see if the cluster's q-score is higher than the expected +1.96·SD value.

  The new measure can only be applied to clusters found by a set of search terms (the probe).

• Added kurtosis and skew measures to the Cluster and Related searches. These are statistical measures of the shapes of the distribution of scores in the found clusters. A high kurtosis (approx. > 3 or < -3) indicates a sharply peaked curve. This means that only a few dreams scored highly for a theme (or by relatedness). A high skew value (approx. > 1.5) indicates a lot of low scoring dreams for the chosen theme.
• Rearranged a few of the reports on the Search panel.
• 03-21-04: New "Quick Start" document gives you quick walk through of some of the basic features.
• New "AutoAdd" checkbox option on the Settings panel. When you start up Bag O' Dreams it will automatically load/add the last analyzed set of dreams (if the box is checked). If you never saved your dreams in a particular folder then you'll be prompted to choose one.
• Closing the little progress window will stop this if you've changed your mind.
• The start-up process now includes an extra step which calculates the cluster statistics for the set of dreams. Having this done at start-up makes the AutoCluster reports a little faster. (Unfortunately it also makes start-up a little slower!)
• This also allows for the inclusion of an expected significance level in the Cluster, Related, Random cluster and Co-cluster reports. These reports now give you some idea of how coherent (single-minded/single-themed) a cluster of dreams is.
• As a small step towards better integration of the editor with the rest of the program I've put an "Add" button on all of the dream windows. (It looks and works pretty much the same as the "Guest dream" button on the Settings panel.)
  • After, you've recorded a dream and saved it in your dreams folder, clicking the Add button will add it to your existing dream set.
  • If the dream is already in the dream set and has been Saved, then the term frequency tables will be updated.
  • Note that you must have already processed (added) a set of dream to Bag O' Dreams for this to work.
  • In some future release I may make it so that a dream's term frequency tables are silently updated whenever you Save it.
• 03-19-04: When you start up Bag O' Dreams it now prompts you to select a folder of dreams for analysis.
• 03-16-04: New registration system personalized by your name.
• Bag O' Dreams now loads any executable "jar" file found in the modules folder at start up.
• Minor changes to the format of the "r-cluster" report.
• 03-10-04: The Mysterium Coniunctionus command now produces a linked chain of random text from a set of dreams. The first word is chosen randomly, along with the next 5 to 15 words in the dream. The last word in this snippet is then used as the first word in the next piece of the chain. The number of links in the chain is taken from the threshold field on the Search panel.
• 03-06-04: Added some regular expression commands ("Regex") to the Search panel and Search menus of dream and cluster result windows. These allow you to search individual dreams, clusters and sets of dreams for a single regular expression. All searches are case insensitive.

  A few examples;

  fl(y|oat){1}(ing|s){0,1}
  matches: "fly", "flys", "float", "floats", "flying" and "floating"

  [^a-c]at
  matches: "rat" and "hat" but doesn't match "cat" or "bat"

  [mf][oa]ther(.in.law)?
  matches: "mother", "mother-in-law", "father" and "father in law" (no hyphens)

  \bdog\b
  matches: "dog" but not "doggerel" or "hotdog"

• The tagging reports now exclude commented out lines.
• 03-02-04: Added some report commands to the Search menu in result windows;
  • Frequency within
  • Dreams within
  • TIC within
  • Neighbors within
  • Pairs within
  • Expand within
  These work just like the reports on the Search Panel except that they only look at the dream(s) within a cluster.
• 02-28-04: New Anti-Cluster report. This one does a Cluster report and then returns all dreams not belonging to the cluster. To use it the first character in you search string must be a tilde followed by a space ("~ "). When you run a Cluster report with this prefix you get the complement of the cluster that would normally be found. I made this report so I wouldn't have to use the technique described in the "Some results" document.
  Some things to remember:
  • Individual document scores are meaningless in an Anit-Cluster report (which is why they're not included).
  • The theme version of the Interpretation report only looks at the highest scoring dreams in a theme. Therefore it has little meaning in a Anti-Cluster of size greater than 10.
  • Set the threshold to -1 if you want to use Anti-Cluster to find all dreams without characters (for example), because this will ensure that the "positive" Cluster includes all dreams with any characters. (See the "Some results" document for instructions on how to construct a search string which will return all characters.)
• 02-26-04: The matching terms returned in both the Cluster and Related searches are now hyperlinks. Clicking on one of these links triggers a terms in context (TIC) search within the dream next to it.
• 02-24-04: Added an indirect look-up feature for the alt.dreams newsgroup.
  Now any Bag O' Dreams window which isn't editable can search alt.dreams for postings by a particular author or on a particular subject.
  To use this feature:
  • Go to the About Panel and open any window from the drop-down.
  • In the text field enter one of the following;
    google:author: john smith
    google:topic: teeth tooth
  • Select the "Find" command from the "Search" menu.
    The first one will give you all posts authored by john smith.
    The second one will give you all posts containing teeth or tooth.
  This feature depends on Google and may not work in non-Windows or non-Internet Explorer set-ups. It's sure to break eventually and so it may not be supported in future releases.
• 02-22-04: Bug fixed: if you tried to open a file while running an Interpretation report you got an error.
• 02-21-04: I've improved the look-up method of the dream dictionary: it's now is better at finding exact matches for a topic.
• New interpretations file available: the public domain (and rather silly) "10000 Dreams Interpreted" by Gustavus Hindman Miller, formatted for use in Bag O' Dreams. To use it click on the link and save it as a text file, replace the interpretations.txt file in your settings folder with the new file (after you make a back-up copy) and restart Bag O' Dreams.
• 02-19-04: Most of the reports can now be run concurrently. Previously I blocked the running of more than one report on each panel: the drop-down selectors were disabled during a search. This was partly to prevent the user from exhausting memory. Now you can run as many reports as you like. The only exception is the AutoCluster panel reports because they use a lot of memory. Note that running many reports at the same time slows performance.
• Removed the "multi-cluster" reports because they were somewhat redundant and I didn't find them very useful.
• Removed the MultiExpand report for the same reasons.
• 02-17-04: Reformatted the tagging reports.
• Changed the name of the "Report" menu to "Search".
• Fixed a bug where the string which indicates negation "(~)" was interfering with "find within" searches.
• 02-15-04: A few more adjustments to the new reports look.
• A few small optimizations to the rendering of some reports.
• 02-13-04: Revised the look of all reports: added color and made layouts more consistent.
• Added "MultiExpand" report. This report reads words, line by line, from the "expandlist.txt" file in the settings folder. It does an Expand report on the words found on each line.
• 02-10-04: Revised the Neighbors, Pairs and Subsumption reports: they all use thresholds to exclude low scoring results. For the Pairs and Subsumption report a default threshold of 0.7 is used if a "bad" value is entered. For the Neighbors report the default window width is now 50. All three of these reports now generate a "Better terms" link.
• New "Expand" report is a combination of the Neighbors and Pairs reports. It generates search string probes from single words or small strings of words. Basically it does a Neighbors report on a single word and uses the Better terms in a Pairs report. The result is up to 20 Better terms which can be used in a Cluster search to see if the dreams returned have any intuitively recognizable themes. Best results are found when you use only the first 7 or 8 terms.
• 02-08-04: Changes to the theme/cluster version of the Interpretations report. Now the scores for typical themes, personal themes and emotion words are weighted by their occurrence in the best 10 dreams of the cluster.
• 02-07-04: Fixed a few small display bugs in the Related report and the r-Cluster report.
• 02-06-04: Several changes to "Find within" methods (formerly called "Search within").
• Reports that return clusters of dreams now have links ([find within]) which trigger a Find in context search.
• Reports that return multiple clusters or dreams (AutoCluster and r-Cluster) now have a "find with" link. Clicking on one of these links pops up a new window with just the dreams in the selected theme/cluster.
• Both flavors of the Interpretation report now have "find within" functionality. All of the terms and phrases found for each theme are now "find within" links.
• The Find button has been replaced by a Report menu in result windows.
• 02-04-04: The "Search within cluster" command now saves "Better terms" in the reports that generate them.
• If you reopen a saved "Search within cluster" report the Find button returns a Terms in context report, but remember that this only works for those dreams that have been added to Bag O' Dreams (i.e. currently active/processed dream documents).
• 02-03-04: New "Search within cluster" command, allows you to search for text within a cluster of dreams. This command shows up in reports which return clusters as link that looks like this: Search within cluster. When you click on it a new window is created which lists the dreams in the cluster, their Best terms and Frequencies. Clicking on these links enters their terms in the window's search text field instead of the main Search panel. The Find button does a Terms in Context search, but confined to the dreams in the cluster.
• 01-31-04: Some minor changes to how the Tags command matches words with suffixes.
• Some updates to the mytags files.
• 01-25-04: Added a "Find and Tag" command for all dream document windows. This finds the words entered in the search field and then color highlights the results in a new window.
• Changed the way the color tags look ("All tags" command);
  Key: Actions Animals Body Characters Emotions Objects Settings
• 01-21-04: Bug fix for "All tags" command: it was generating bad HTML which only showed up if you saved the tagged report and viewed it in a web browser.
• 01-20-04: First draft of Objects and Actions files in the "mytags" folder. Use these to color tag common objects and actions in dreams.
• Added "control-f" to Find in all result windows.
• 01-18-04: Changed the theme version of the Interpretation report. It now shows links to topics in the dictionary instead of the full text of the definitions.
• Added a section on "How to record your dreams" to the documentation and revised the "Typical themes" document.
• 01-12-04: Added new searches for all items in the Search menus. These "multi-cluster" items show up in the left drop-down on the "Search panel". When you select any of them a quick cluster search is performed for each menu item. This shows you the percent of files with each "theme" and the (up to) five highest scoring dreams for each theme. (Note that not every menu item necessarily defines a theme, so some of the results won't be too interesting.) This replaces and extends the old "Typical themes" and "Personal themes" reports.
• Renamed the old "Characters" report to "Names" since "Characters" is the name of a menu.
• Beefed up some of the search strings under the "Emotions" menu for the new report.
• Replaced the drop-down selector in all dream windows with a menu named "Report". The "Find" button was also removed and put in the same menu. (Remember that you can type "control-f" to Find text.)
• 01-10-04: Updated the "My Tags" report. I've renamed it "All tags" and added separate tagging reports for the individual files in the "mytags" directory. If you select "All tags" the words and phrases in all of the mytags files are highlighted in the current dream. If you select only one file, only those terms and phrases are highlighted.
• Removed the old "Color terms" document report because I seldom used it.
• The "Theme tree" report (an optional part of the AutoCluster report) was removed because I didn't find it of any use.
• The undocumented and unused "Tags" buttons were removed from the document viewer.
• 01-08-04: Most of the drop-down selectors now display more items than they used to.
• The Subsumption report (on the Search panel) now accepts a threshold parameter which must be between 0.0 and 1.0. Previously the parameter was hard coded at 0.7. A lower threshold will cause more terms to be included in the results.
• The menus, interpretations and documentation files have been revised.
• 01-04-04: Changed the rule for cluster quality (q-score) so that smaller clusters of dreams get a lower score. This results in "clusters" with only one dream being given a score of zero. The change is most noticeable in the AutoTheme report.
• 01-01-04: Changed the User menu to be just like all the other menus on the Search panel. This means you can edit it with MenuEdit or with the "Save terms" command.
• 12-29-03: Fix for bug in Interpretation report where the wrong text was sometimes displayed for a theme.
• 12-24-03: The Dream Dictionary is now editable from within Bag O' Dreams. A little drop-down on the Dictionary panel contains three commands; "New", "Save" and "Delete", which do what you'd expect.
• The format of the "interpretations.txt" file (used by the dictionary) has changed. If you've already edited this file then you'll have to change your definitions to the new format.
• 12-18-03: Changed some code to handle low memory situations better.
• Changes to documentation and removed some unlikely names from the "names.txt" file.
• Changed a few "documents" references to "dreams".
• 12-12-03: Added progress indicators to "My tags" and "Color tags" reports.
• Added count of dreams by month and by day of week to the "Docs info" report.
• 12-11-03: "My tags" no longer substitutes the stem of a word for the word found - it uses the actual word found. Also, added some extra formatting (underline and italics) to help distinguish categories.
• 12-10-03: New "My tags" tagging system for common items of interest. When you open a dream document the drop-down selector will display "My tags" if you've put any files in the new "mytags" directory. These should be plain text files with a single word or phrase (no quotes) on each line. When you select "My tags" the current dream will be tagged wherever it contains items from the lists. A "key" is also displayed which shows the color assigned to each item. The names of each item are taken from the names of the files in the mytags directory:
  Key: Animals Body Characters Emotions Misc Settings
  I've included a few files to get you started. In particular you should add all known characters to the Characters.txt file.
  Note that "My tags" will find most forms of a word given a stem: "play" finds plays, playing and played; "nervous" finds nervously, nervousness
• 12-06-03: New "Terms from file" command lets you enter a set of search strings from a plain text file. The file should have each term or phrase on a separate line. When you run the command it prompts you to select a file, then it tests the current set of dreams for occurrences of the terms or phrases and enters any found into the Search panel's text field. You can test this new command with the "slang.txt" file. "Terms from file" is located in the lower right drop-down on the Search panel.
• 12-04-03: New "Characters" report which searches a set of dreams for over 5000 common names.
• 12-03-03: The "Update menus" command has been renamed "Save terms". Selecting it saves the currently entered search strings (words and phrases) with the last selected menu item.
• 11-28-03: The Neighbors report now generates a "Better terms" link instead of a list of every neighboring term.
• The Co-occurrence version of the cluster report now has the same format as that of the Search panel.
• Removed the underused Rare terms report.
• 11-23-03: Fixed a bug that sometimes excluded a word from a dream's term table if the plural for of the word was already in the table.
• 11-21-03: Small change to the Co-occurrence command which copies search strings from the Search panel to the Co-occurrence panel. Now it looks to see if either text field is empty and puts the string in an empty field if possible.
• Cleaned up some code - no changes that the user can see.
• 11-18-03: Removed "document weighting" option from AutoCluster report.
• 11-17-03: Now using Steven Spencer's BrowserControl class to display web documents in the user's browser instead of in a Bag O' Dreams' window. This allows you to display HTML and PDF documents. This works for Windows and UNIX only.
• 11-15-03: A somewhat hacky fix for display of images in HTML documents.
• 11-14-03: Added minimal support for HTML hyperlinks. Pages load very slowly. Only very simple HTML documents are supported: no support for relative paths to images, framesets, javascript, stylesheets. To test links out try the new References document (in the drop down selector on the About panel.)
• 11-08-03: Added "Theme interpretation" report to the "r-Cluster" report and the "Cluster" report on the Co-occurrence panel.
• The Find command in document windows now wraps around (starts back at the top) when nothing is found.
• HTML documents placed in the settings folder are now loaded in the drop down list on the About panel. (Their names must end with ".html".)
• 11-07-03: New "Theme interpretation" report. This one shows up as a blue hyperlink (Interpretation) in the following reports;
  • Cluster report (on the Search panel),
  • Random Cluster report (on the Search panel),
  • Relatedness report (run from individual dreams) and
  • AutoCluster report (on the AutoCluster panel).
  If you click on the link it runs an Interpretation report which treats all of the dreams in the cluster as one big dream.
• 11-01-03: New "modules" folder for stand-alone/optional widgets that are included with Bag O' Dreams. Right now these include;
  • The MenuEditor,
  • The RPN calculator,
  • The h-score calculator and
  • The thesaurus (available as a separate download).
  These should be placed in the new "modules" folder which must be in the same folder as the bag.jar file. When Bag O' Dreams starts up it looks in the modules folder and loads the items found in the second drop down selector. (That's where the two calculators were accessible from before.) This will allow you to drop in new modules or leave out ones you never use. I might make an infix calculator (drop-in replacement) for those of you who don't like the current RPN calculator.
• 10-31-03: Added a "Tree" checkbox to the AutoCluster panel. If you check it a "theme tree" is generated at the bottom of the AutoCluster report. If it's not checked the tree isn't added. I added this option because the theme tree takes a lot of time to generate and I haven't found it very useful.
• Added a document weighting checkbox ("wt") next to the AutoCluster button. If you check it the AutoCluster report uses a different weighting scheme for term frequencies. This is "experimental" and likely to change in future builds.
• 10-27-03: Changed the layout of the "Dream dictionary" panel a little and added a "look up selected text" feature to the text area: clicking the Find button will look up whatever text is selected.
• 10-23-03: New "Add guest dream" button on the Settings panel allows you to add a single dream to an existing set of dreams.
• The "Dream dictionary" that's included with Bag O' Dreams is now available on-line (from a link at the bottom of the section describing the dream dictionary).
• 10-20-03: Revised the "interpretations" file and the "References" section of the "Quick Guide". Also updated some of the default search strings.
• 10-18-03: The "Correlation" report on the "Co-occurrence" pane has been revised;
  • The statistical sampling method now uses the product of the left and right scores instead of their difference. This gives greater weight to cases where both the left and right scores are high.
  • Added a new caution message for cases where the occurrence of a theme is very low (less that 5% of dreams).
  • Added a second caution message for cases where the quality of a theme is very poor (any "theme" that occurs in more than 20% of dreams).
• 10-16-03: Added a free Thesaurus application - see link on main page.
• 10-14-03: Fixed a minor bug where cluster with scores of less than 1.0 didn't display their "Better Terms".
• Added link to frequently asked questions (FAQ) on Bag O'Dreams page.
• 10-06-03: Increased the number of random samplings used in determining the statistical significance of some results.
• 10-01-03: Price cut to $20 USD (was $30).
• 09-23-03: "Terms only" option on AutoCluster panel: hides display of documents in the automatic clustering reports.
• All cluster reports now display terms found as a link. Click on link and the terms are entered into the Search panel.
• 09-19-03: Fixed bug in AutoTheme report where the number of "stop" iterations was not being recorded.
• The Correlation report now highlights individual document scores that are higher than the mean.
• 09-08-03: New plus/minus button on the Search panel will toggle a prefix on all search terms and phrases; plus sign then minus sign and then no sign.
• The RPN calculator now has a keyListener so that you can use it from your keyboard.
• 09-06-03: Small optimizations to the r-Cluster report - about 20% faster.
• 09-01-03: New "r-Cluster" report in the AutoCluster panel. This one clusters dreams based on relatedness of terms. Dreams are clustered together if their relatedness is greater than or equal to the value entered in the "r-threshold" field.
• 08-14-03: AutoCluster now produces a hierarchical tree showing the relatedness of the themes found. This "Theme tree" shows closeness/distance via nested parenthesis: closer items having no or few parenthesis between them.
• 08-05-03: AutoCluster now has an automatic stop condition. If the increase in overall cluster quality is very small then iterations stop.
• The initialization of the clusters now uses one randomly selected dream per theme. The remaining dreams are them added to the cluster to which they are closest.
• I've added a little move verbiage to my "Typical themes" rant.
• Repackaged a lot of things - nothing that the user can see.
• 07-24-03: The Interpretation report now indicates if a term or phrase was negated: terms and phrases that are preceded by "no, not, isn't, can't,..." are indicated by a '(~)'. This can help distinguish between "he helped me" and "he wouldn't help me", both of which score the same in a Cluster search for "Friendliness".
• Cleaned up the Split and Concatenate commands.
• 07-18-03: Dreams found by AutoCluster are now ordered by similarity to the set's terms vector.
• Increased the maximum number of dreams from 50 to 100 in demo version.
• Fixed a bug in the dream dictionary which sometimes caused the display of the wrong definition for a topic.
• 07-12-03: "Interpretation" report now shows a little more information for emotions, interactions, success and misfortune.
• Minor reformatting of "Pairs" report - made it wider.
• Fixed a few minor bugs with display of documentation and moved build history to its own document (this document).
• 07-10-03: New version of the "AutoCluster" report forces each dream to belong to one and only one cluster. Most of the old parameters are set automatically. This produces a smaller set of results with no duplicate clusters.
• New "Correlation" report in the Co-occurrence panel. This runs two Cluster reports (using the left and right search terms) on each dream and determines if there is a correlation between the two themes.
• Cluster searches now give some weight to terms outside of the minimum and maximum document occurrence setting. The idea is to replace the previously used sharp cutoff with a gradual ramping of the weight given to terms that are too rare or too common.
• New "Subsumption" report. This one shows items that rarely occur unless other terms are present in the same dream, hopefully showing hierarchical relationships between terms and phrases.
• The Co-occurrence panel was redesigned to use a selectable list of reports rather than buttons.
• 06-15-03: New "Cluster/Frequency" report. Shows all dreams in chronological order and displays cluster score for the current search terms and phrases. Also shows the terms and phrases that were found in each dream and their frequencies. Useful if you want to see how often a theme, phrase or term occurs over time.
• The "Interpretation" report now shows highest scoring themes first. Also uses the threshold field (on the Search panel) to exclude low-scoring themes.
• The "Interpretation" report now has a "see also" look-up feature. If a line in the dream dictionary contains the phrase "see also" and a topic in quotes that has a definition, then a link to the dictionary is created. Clicking on the link looks up the topic in the dictionary.
• The "Neighbors" report now shows frequency of terms found.
• 06-06-03: Added "Dream dictionary" with about 250 basic topics. Redesigned the "Interpretation" system to use a single file instead of two. It now uses the same file as the dream dictionary. Redesigned some classes to reduce memory requirements.
• 05-29-03: Redesigned the three co-occurrence reports and added the ability to search by phrases as well as single terms. The "Neighbors" and "Pairs" searches now understand phrases too.
• 05-23-03: A small change to the cluster reports was made: now a document that only matches a few search terms or phrases is scored slightly lower than one that matches several. A bug affecting some searches by phrase was found and fixed.
• 05-18-03: Added term "Frequency" list to all cluster reports. This shows the frequency of each term in the result. Also changed the "Interpretation" report again. It now displays the five closest term-related dreams.
• 05-12-03: Major overhaul of the search menus. Added stand-alone "MenuEdit" utility: GUI for editing all search panel menus (except the User menu). See the section titled "The Search menus" for details.
• 05-05-03: More changes to the format of the "Interpretation" command. A bug in the "Update menus" command was fixed: the User menu's search strings were being wiped out.
• 05-02-03: The Interpretation command now shows matched terms for each theme found. It also shows the dream's body-related terms and dream's setting.
• Removed the Typical themes and Personal themes as these are shown in the Interpretation results.
• 04-29-03: An "Interpretation" command has been added which provides interpretations for themes found in a dream. This looks at both the typical themes and the user-added personal themes. Terms that indicate emotions, interactions and characters are also listed in the report. Interpretations can be edited in the /settings/interpretations.txt file. This is just for fun - don't take these interpretations too seriously!
• 04-22-03: A case-insensitive "Find" command is now available in all result windows.
• 04-11-03: "Color terms" command shows a dream's significant terms in color. The colors are arbitrary but words with the same stem get the same color: laugh laughs laughing. This shows you the terms that will be included in cluster and relatedness searches.
• Bug fix: the document selector was not getting filled in when the "Number of dreams to add" setting was greater that the number of dreams available.
• 04-06-03: Option to show only significant scores in automatic cluster searches.
• 04-03-03: Now displays expected q values for cluster scores.
• 04-01-03: You can now add a specific number of dreams, selected randomly from the chosen directory. This is helpful with large sets of dreams where you might not want to add all of the dreams.

Bag O' Dreams      
(http://kitchen-sync.com/bag/bag.html)