I am no longer working on Bag O' Dreams. Instead, please try my Bag O' Regex project which has most of the same functionality of Bag O' Dreams.

Bag O' Regex is available at: http://kitchen-sync.com/bag/BOR.html.





Bag O' Dreams

Bag O' Dreams is an application for the exploration of dreams. Bag O' Dreams allows you to search dreams for themes and content using information retrieval techniques. You can enter a list of terms and phrases associated with a concept or theme and Bag O' Dreams will automatically identify the dreams that are most likely to be relevant to it. You can also start from a particular dream and find other dreams that closely match it in content or generate a list of common themes contained in the dream. Starting at the level of individual words, you can discover which other terms are commonly associated with it, how often it occurs and display it in the context of your dreams. Starting from a collection of dreams, Bag O' Dreams can automatically partition these into sets of dreams that are thematically related.

Bag O' Dreams is intended for anyone interested in the study of dream content.

Bag O' Dreams is a Java application that requires JavaTM 2 Runtime Environment 1.5 (or higher) to work. (See the Installation section below.)

Contents

Installation

  1. If you don't have a Java Runtime Environment (JRE), go to the Sun site http://java.sun.com/getjava and install one.
  2. Click this link to download Bag O' Dreams.
  3. Unzip the file and double-click on the bag.jar file to launch Bag O' Dreams.
  4. If you are upgrading from a previous version, don't forget to back up any text files that you have modified in the settings and mytags folders such as the menus.txt and interpretations.txt files. (Usually the new parts of a new version are the bag.jar file and the HTML documentation.)
  5. Still having problems? See detailed set up instructions (for Windows) in the Frequently Asked Questions document.

Important: Bag O'Dreams requires version 1.5 (or higher) of the Java Runtime Environment to work. Download this from Sun at the link above.

Basic features

Settings panel

setting panel

The Settings panel is where you add a set of dream documents that you'ld like to study. Settings control how terms are added to the term table. Usually you'll want to exclude words that only occur in a handful of dreams and words that occur in almost all dreams. What remains are the words which carry most of the dreams' content.

Add Directory

Click on the add button folder: add button to navigate to the folder where you keep a set of dreams. Click on any dream in the folder and the whole set of dreams is added to Bag O' Dreams. The dreams must be .txt files with one dream per file. Once added, you will be able to run any of Bag O' Dreams' reports on the set of dreams.

The "Guest" dream

When you only have one dream from a dreamer and want to quickly scan it for themes, add the dream to an existing set. Then you can run various reports on it - especially the "Interpretation" report. The "host" dream set can be your own dreams or any collection of dreams which you'd like to compare the "guest" dream to. The guest dream button is also helpful when you've already processed a large set of dreams and only want to add a dream which you've just finished recording.

The guest dream button does nothing if you haven't already added a set of dreams.

Newly added dreams appear at the bottom of the document selector list.

You can also add a single dream to an existing set by saving it in the same folder as the set. When you do this Bag O' Dreams will ask you if you want to add it to the current set of dreams.

Search panel

search panel

The Search panel provides many methods for finding text and identifying themes in your dreams. The Search Menus store lists of terms and phrases which identify specific content in your dream set. When you select an item from a search menu, like Driving, a list of related words is entered into the text field. More details here on editing the Search menu lists. Reports which use search terms are described in this list;

A second drop down selector lists reports that give global information about the set of dreams and which don't depend on search terms;

A third lists reports that give global words used in the set of dreams;

A forth one lists the drop-in modules and a few utility functions;

[*These are optional stand-alone applications that only show up if they are put in the "modules" folder.]

The small combo next to the search terms field provides a few commands that effect the search terms. search string commands

Save: Save the current search terms to the last selected menu item.
Expand: enter one or more words or phrases and find the words most commonly associated with it.
Lemmatize: remove words which have the same stem form the list.

Document-specific searches

To create a new dream document click on the "new" button: new.

To open a dream via the standard file dialog, click on the "open" button on the Search panel: open.

You can also use the document selector in the middle of the search panel to open a dream for editing or analysis. When you select a dream from this list it opens in its own window. You can edit the dream and run the following reports from the dream's Search menu;

In addition, you can do simple text searches inside the dream with the Find command (Typing "control-f" also works.).

To tag (color highlight) all search terms at once use the Find and Tag command.

Co-occurrence panel

co-occurrence panel

This panel provides a way of searching for terms in context where the order of the terms is important. The "Left" and "Right" term fields and their "threshold" fields allow you to distinguish between "dog bites man" and "man bites dog".

The User menu can be customized by editing the file "co_usermenu.txt". Use the format:
  1. Menu item title
  2. Left terms
  3. Right terms
Example:
  1. Feet/Foot
  2. my her his its their
  3. feet foot heel toe

This example is used to distinguish "feet as body part" from "feet as measure".

To insert a separator between two menu items just include a line with the string "separator" in the file. To leave a text field empty insert an empty line.

Selecting the "Edit" item in the User menu displays the co_usermenu.txt file in dialog text area. After you have edited it you can save your changes be clicking the "Save" button.

Co-occurrence reports

Automatic cluster detection

autocluster panel

Bag O' Dreams uses four methods for finding themes in a set of dreams. These are; k-means, k-fuzzy and r-Cluster. These are collectively referred to as AutoCluster reports.

k-means

This is a method for finding themes in a set of dreams without user-determined sets of search strings. This makes it a "blind" search method. K-means uses a variation on a technique described by Dhillon and Modha (i.e. the spherical k-means algorithm) to find the themes.

The algorithm starts by randomly partitioning the dreams into some number of sets (usually 10 to 40). The quality of each set is determined and a concept vector created. (The concept vector is a numerical representation of all of the words and their frequency of occurrence in the set of dreams.) Once this step is done each dream is compared to each of the sets of dreams and moved to the one to which it is closest. Next, the sum of the set's quality is calculated and the steps are repeated until this sum stops increasing.

To use automatic cluster detection follow these instructions

  1. "Add" your set of dreams with max doc occurrence set to about 15% of the number of dreams in your set.
  2. Select the number of clusters you'd like to find.
  3. Click the "AutoCluster" button.

After a long time you'll see a table of results consisting of lists of dreams, their "best terms", "characteristic terms" and a measure of the quality of each cluster generated. Clusters of 5 to 20 dreams seem to be the most coherent.

For my own dreams I get some intersecting results. My flying dreams include the terms "fly, float and glide" as expected, but also "ball, body, fence, tuck, ability, inch, land". I am sometimes flying over a ball game of some sort, or over a fence. I also tuck into a ball. In many cases I mention the position of my body while flying and reflect on how unusual my ability to fly is. Often my flight is only an inch above the ground and I often wonder where I will land.

Other themes that often show up are "being on or near a beach", "swimming" (distinct form just walking near a beach), being at a "train station/trying to catch a train", "trees", "climbing" and a second slightly more narrowly defined variation where I climb on a hillside that is loose or crumbling. I've also found "my old laboratory workplace", "coin operated machines/finding money", "apartment living/secret rooms" and "boy in danger of falling" themes. A very clear "reading" cluster also show up, though I have very few dreams about reading. Maybe this is because my reading dreams are only about reading.

k-fuzzy ("Fuzzy" k-means)

The k-means algorithm only allows a dream to be assigned to a single cluster - the one that it is "closest" to. Since many dreams contain more than one theme I thought it might be useful to reassign each dream to the cluster where it scored second best. The Fuzzy k-means report lists each dream in two clusters; the dreams found by non-fuzzy k-means and "secondary" dreams which score higher in other clusters but also might have been included within the primary cluster. The first dreams in the secondary cluster often score higher than the last dreams of the primary cluster!

Tip: Since AutoCluster reports take a long time to run consider saving them as HTML files in the "settings" folder. When Bag O' Dreams starts up it loads any HTML files that it finds there into the drop-down selector on the About Panel. If you've reloaded (added) the same set of dreams you can start where you left off in your exploration of the report's results.
"Sharpening" clusters

The k-means algorithm assigns every dreams to a cluster. In some cases a dream might be so different that excluding it from all clusters would improve the coherence of the results. You can exclude poorly fitting dreams from k-means and k- fuzzy results by entering a "closeness" threshold value (in the r-threshold field). A good starting value is 20 (range > 0 to < 100). When you run k-means or k-fuzzy, dreams that aren't close to any cluster are excluded. These dreams are displayed in a list at the bottom of the report titled "Outliers".

r-Cluster

The "r-Cluster" report is found in the AutoCluster panel. This report clusters dreams based on relatedness of terms. Dreams are clustered together if their relatedness is greater than or equal to the value entered in the "r-threshold" field. This is the only parameter used by the report and should usually be set between 10 and 30. A low threshold results in several very large clusters (near the end of the report). A high threshold tend to result in a set of smaller clusters. When the "r-Cluster" button is pressed a relatedness matrix of all of the dreams is generated. Next a single dream is randomly selected and all of the other dreams with relatedness scores at or above the threshold are added to form a cluster. The process is repeated until every dream has been added to a cluster. (Note that this process gives greater weight to the first dreams selected so that by the end a large number of dreams end up by themselves in single-dream clusters. Even so, these dreams tend to have fewer close neighbors than dreams in larger clusters.) Set minimum cluster size to 3 or 4 to avoid seeing the very small clusters.

"Spotlight" dreams

The phrase "spotlight dream" comes from Calvin S. Hall. He used it to mean a dream who's meaning was obvious and helps to illuminate the meaning of other dreams in a set. In this context spotlight dreams are simply the highest scoring dream in each cluster. These dreams ought to give a good idea of the main themes in a set of dreams.

The "Spotlight" dreams link appears near the bottom of the AutoCluster report.

Index

The "Index" report is found in the AutoCluster panel. After you run at least one AutoCluster report (k-mean, fuzzy-k or r-cluster) an "index" is generated. The index is the intersection of each cluster's characteristic terms and the top scoring dream (spotlight dream) of the cluster. The report shows the list of spotlight dreams with their index terms. The idea is to make the characteristic terms more concrete by filtering them through an actual dream document. As you run more AutoCluster reports more terms and dreams are added.

Dream dictionary

dictionary panel

Dream dictionaries are notorious for producing nonsensical, silly or just plain obvious interpretations for various elements occurring in dreams. Even those who believe that dreams have deeper hidden meanings tend to scoff at the use of dream dictionaries because they claim that dream symbolism is individualized to the dreamer. Those who are convinced that dreams are random nonsense don't feel there is a need to explain every detail of a dream because dreams aren't messages at all. While I agree that dreams aren't coded messages (e.g. from the unconscious) I think that dreams are none the less meaningful in the following ways:

Anthony Stevens points out that some of the nonsensicalness of dream dictionaries stems from translations which obscure puns that are only apparent in the original language.
"... to dream of uncovering one's behind means one is about to lose one's parents. This is incomprehensible unless one knows that the Egyptian word for buttocks closely resembles the word for orphan."
- Private Myths, p.15.

The claim that (apparent) symbolism found in dreams is all made up in the process of a "dream expert's" interpretation, like a "cold reading", or that holding a particular "theory of dreams" predisposes the dreamer to have or recall dreams that match the expectations of the theory has some merit. But in spite of this I'll continue to identify some dream elements as symbolic even though I don't think there's a agent working behind-the-scenes, trying to send me a message or trying to obscure or censor the "true" content of my dreams.

The usual assumptions made by a dreamer, while dreaming, are that they are awake, embodied and situated in a place and time. These assumptions make up a context for the dream which acts as a filter on the incorporation of internal and external stimuli (or the lack of them!). Under these conditions, the call of a cardinal (a very red member of the finch family) which eventually wakes me is symbolized as blocks of red colored text periodically scrolling by on a computer screen when the ongoing context of the dream was "searching for a bug in the code of a computer program".

"I am looking at some java code, trying to find a bug. It scrolls by in a computer window (or just a paper print out?) Every so often I come to some code comments that are printed in red. I slowly wake up to hear a cardinal singing: its bursts of song are spaced at the same interval as the red code comments."

In another dream the ongoing context of being at a workplace was interrupted by an episode of sleep apnea. This was symbolized as "squeezing past an annoyed man in a narrow corridor". Hartman gives an example of a woman with a sore throat who dreamed (symbolically) of a staircase with pink scratched walls (ongoing context not mentioned). Though anecdotal in nature, these cases show that elements of a stimulus are only incorporated in a dream if they match its ongoing, assumed context. Other elements that don't match context never reach the level of conscious awareness.

In the case of the cardinal dream, there's no reason to think that "my unconscious*" acted as a censor to my becoming fully aware of the fact that a cardinal was singing (*i.e. my unconscious as an agent the "per-screens" all sensory data and only lets me be aware of some of it - to preserve sleep, as a Freudian might claim). Instead, I'd prefer to describe this dream as a result of automatic "feature detectors" working to find associations for the bird song. These "feature detectors" would have normally been able to recruit and activate memory and attention to the fact that a cardinal was singing, however, the ongoing dream and my conviction in it's reality, needed to be overcome first. In the mean time, only "red" and "periodic" make it to conscious awareness (because these features could be fit to ongoing context).

A dream dictionary can't know the ongoing context of the dream. In this way the objections that the meaning of a dream symbol is too individual to be simply looked up on a list is seen to be well founded. It is unlikely that I would have guessed that a cardinal's song was the source for the periodic appearance of red text in my dream if I hadn't woke up and heard the bird singing. I'm comfortable calling "red/periodic" symbolic of a cardinal singing even though there's no behind-the-scenes agent who's trying prevent me becoming aware of the bird singing, disguising and censoring it.

An unintentionally ironic entry from "10000 Dreams Interpreted" by Gustavus Hindman Miller might serve as a admonition against relying on dream dictionaries:
"To dream that you are referring to a dictionary, signifies you will depend too much upon the opinion and suggestions of others for the clear management of your own affairs, which could be done with proper dispatch if your own will was given play."

So why a dream dictionary? Well, for one thing, some people believe that dreams have deep, hidden meanings and that elements that occur in their dreams are symbols. They couldn't care less about my opinions on dreams! For these users it's fairly easy to write their own list of symbols and definitions. (Use the little drop-down menu to add/edit or delete dictionary entries.) The second reason is that the Interpretation report uses the same file as the dictionary to provide suggested meanings for themes found in a dream. These can include both traditional, metaphorical interpretations as well as "brain based" explanations for common dream phenomena. Since a dream can contain more than one of these themes I though it might be interesting to see if the list of "meanings" provided by different themes intersects in a dream. Maybe a "car out of control" dream is just a variety of a "flying" dream with all of the motion confined to the horizontal. Maybe the difference between the two varieties of dreams is a tacit awareness of the sleeping body's horizontal position - with this (and "fictive motion/REM motor cortex activation") you get "flying", without it you get "car out of control".

The Dictionary has a simple built in editor (controlled by the drop-down with the tiny icons). There are a few limitations with this;

Here's an online version of the dream dictionary that's included with Bag O' Dreams.

You might also find the Cambridge dictionary of idiomatic expressions useful.

Hyperdictionary also provides this excellent dictionary/thesaurus:
Dictionary Search
Search for:
Search by Hyperdictionary.com

For a frankly nonsensical dictionary see Gustavus Hindman Miller's 10000 Dreams Interpreted. This is also available in a format for use in Bag O' Dreams. To use it click on this link and save it as a text file, replace the interpretations.txt file in your settings folder with the new file (after you make a back-up copy) and restart Bag O' Dreams.


Typical use

How to record your dreams

How you record your dreams is up to you and depends on why you're using a dream journal. I am interested in all types of dreams and want to record as many details of each dream as I can remember. I tend to not record a dream if I feel like I can't recall some large portion of it. In general, I think you need to record setting (including physical lay-out), other characters, emotions and thoughts. Only record what happened in the dream - don't include thoughts and associations as part of the dream (except as comments attached to the dream) unless they actually occurred to you during the dream. Another potential pitfall is recording elements that should have been in the dream but which you don't explicitly remember. An example of this is dialog. In my own dreams I can seldom remember exact dialog, even when the words spoken seem called for in the dream, even when I react as though the words were spoken. In these cases I indicate that I'm paraphrasing:
"He said something like 'blah, blah, blah'."
If I'm not sure of an event or element I put it in parenthesis and add a question mark at the end:
"We drove in my car(?) but got to the airport too late."
A lot of popular dream interpretation emphasizes emotions in dreams and asks the dreamer to give emotional impressions for almost every element in a dream. This seems a bit forced to me, especially as events in dreams are sometimes experienced without a lot of emotion. Adding emotion to a dream when there really wasn't any distorts the dream to fit the theory.

The cluster, relatedness and theme searches used by Bag O' Dreams work best when you have a fairly large collection of dreams. Detailed dream reports are needed in order for these methods to reveal any unexpected relationships between dreams and dream elements.

Dreams fade quickly. If possible try to go over the dream before you get out of bed. Record the dream as soon as possible. Write (on paper) as many details as you can remember. Get the overall sequence of events in order:

Starts out at my parents' house,
Joe is there, strange red dog,
My brother, my father, my sister, my daughter,
waiting for someone (my mother?) to return,
I go out and drive to the beach,
two women in a car,
follow them to a pier,
a child in a small boat (I'm very concerned he'll capsize)
etc.
Fill in details:
Joe (friend from work) sat to my right at long table in dining room. It was twilight.
He said something like "when do we eat?",
There was a small dog with red fur that sat curled up under the table. I was afraid that I'd put my foot down on its tail.
We were waiting for my mother (I think?) to return so we could start the meal.
The occasion wasn't really specified - might have been my son's birthday(?)
Nothing much seemed to be happening for a long while, just feeling a little bored
I finally decided that I'd go out and try to find her. Its very dark now. I get in my car.
etc.

Select a title: something unique about the dream or a phrase from the dream. This will make it easier to remember the dream later.

Select a date: I usually use the date that I recorded the dream, assuming that the dream occurred after midnight. (Use the "Date stamp" command in the File menu.)

Save the dream in your dreams folder and you are ready to add it to Bag O' Dreams.

Adding a set of dreams

Bag O' Dreams requires that dreams to be recorded in plain text files (one for each dream) in a single folder. Once you have a set of dream documents and have added comment markers where needed (see the Document format section) and have all of your dreams in one folder you are ready to begin exploring your dreams.
  1. Click on the Settings panel tab and adjust the settings to reflect the total number of dreams you are analyzing. The Max setting should be set at 10 to 20 % of the number of dreams.
  2. If the AutoAdd box is checked then Bag O' Dreams will automatically select reasonable Min and Max doc occurrence settings.
  3. Click on the Add directory button and select the directory where your files live. Select a dream file and choose open. All of the dream files in the folder will be processed and added to the term table.
  4. A progress bar will pop up while the dreams are being read into the table.
  5. When this is finished you can use the Search, Co-occurrence and AutoCluster panels to identify terms, phrases and themes in your dream documents.
  6. You can also use the "All tags" command to highlight characters, settings, emotions and other elements in an individual dream.

Document format

Bag O' Dreams works with text files only. These are files with names that end with ".txt". The following comments are used to exclude text within a dream form inclusion in the tables.
  1. # - lines beginning with a single pound character are ignored.
    Typically the first line of a dream is a comment that includes the title and date of the dream:
    "# The old house 06-27-1994"
  2. ## - lines beginning with two pound characters are ignored.
  3. ### - all lines following a line beginning with three pound characters are ignored.
    This can be useful if you keep your comments or analysis at the end of each dream and separate from the actual dream account.

I had the web site of Schneider and Domhoff (reference 1) in mind when adding this feature. If you generate a random sampling of dreams from this site's search page, it is supplied as a long HTML document that can be saved as text. Each dream in the text document starts with a string like the following: "# 001 (01-01-1964)". The "Split" command will nicely chop up this document into it's individual dreams.

Bag O' Dreams also saves the date if it is found in the first line of a dream. This is used for searching by a date range. The format is month-day-year e.g.: 06-27-2002.


Reports in Bag O' Dreams

Reports are windows displaying the results of a search. These can be saved as html documents. Reports contain any or all of the following;

You can "bootstrap" the reports by using the results from one report as input for another.

An example would be to do a 'Cluster (tw)' search using the Driving search menu item;

"drive car ride passenger automobile truck traffic highway steer accelerate van suv".
This should return a cluster of dreams that have an above average amount of driving. It also returns a list of "Best Terms" (words that occur in all or most of the dreams of the cluster). The Best Terms will usually not be identical to the original search terms;
"drive road hill path"
Using them in a second Cluster search often results in a more representative cluster of driving dreams. You can then run reports such as Modes, Interpretation, Quick Profile or Most Significant Terms from the cluster's window to see which themes and features are over or under represented in the driving dreams.

Some example reports are shown here.


Details of selected features

Interpretation

The Interpretation report identifies user supplied themes found in dreams and clusters of dreams. These themes are lists of words and phrases that (if adjusted properly) identify topics in the manifest content of your dreams. These are entered in Typical and Personal menus on the Search panel. Examples are "Flying", "Falling" and "Car out of Control". When the Interpretation report finds a theme it looks up an interpretation in the dream dictionary and includes it (or a link to it) in the report. The Interpretation reports use the same method as the Cluster report to score the occurrence of each theme in a dream which gives you some idea of how relevant it is. It also finds the closest dreams (using term relatedness scores) and lists any words related to emotions, body, characters and settings.

To run an Interpretation report on a single dream, select "Interpretation" from the dream's Search menu.

Theme interpretation

The "Interpretation" report is also available on reports that return a cluster of dreams in their result. If you select the "Interpretation" menu item it runs an Interpretation report which treats all of the dreams in the cluster as "one big dream".

Modes

Modes is similar to the Interpretation report. Instead of using the cluster score of each theme it uses the term significance (based on observed versus expected frequency of terms). Modes shows which features occur in a statistically unexpected frequency within a cluster of dreams. Unlike the Interpretation report, Modes can also show which features are notably absent from a dream or cluster.

The Modes report is available from the search menu of individual dream documents and clusters.


The Search menus

Once the program is running you may wish to customize the menus in the Search and Co-occurrence panels.

The Search menus can be edited by changing the menu.txt file in the settings folder. You can use the "Save terms" command (in the Search panel, right bottom drop-down selector) to edit the currently selected search strings for a menu item. This updates the search strings but will not add new items. To add new menu items use the MenuEdit utility ("MenuEdit.jar" in the modules folder). Changes made by MenuEdit aren't available until you restart Bag O' Dreams.

If you want to edit the "menus.txt" file by hand use the following format;

<Menu title>
menu item
search terms and phrases
Example:
<Typical>
Climbing
climb "almost vertical" "nearly vertical" "road climbs" cliff hillside steep

Avoid spaces at the end of lines. Don't use parentheses, quotes or other punctuation characters in menu titles. Do not use line returns in the search terms text area.

Some useful search strings

Here's some useful strings of terms and phrases to use in various searches.
Metamorphosis
"turn into" instead transforms "changes into" "becomes a" "becomes an" "becomes the" "switch into" "switch to" "now a" "is now" "then i see that" "later is"
Explanations
explanation assume because cause confabulation convince foreknowledge information realize remember unconvincing "find out" "i'm told" "tells me" "must be" "for some reason" "reason why" "ad hoc"
Counterfactuals
"could have" "might have" almost "shouldn't have" "should not have" "if only" "if I" "what if" "wish that"
Real/Artificial
artificial bogus counterfeit facade odd strange fake forged genuine imitation model natural plastic pretend real sham simulated synthetic "supposed to be"
Private/Public
private public access allowed avoid concealed cult enclosed entrance exit exclusive fence forbidden gate guards guest hidden laws members municipal resort restricted rules secluded section tourist trespassing unrestricted visitor "flimsy door" "flimsy lock"
Micropsia
miniature micro microscopic tiny "scale model"
Orientation
"my left" "my right" "behind me" "behind my back" "over my shoulder" up above down under below behind forward follow high low front back beside after
Proprioception
balance dizzy drift fall float fly gravity levitate lifted skid slide slip swerve swing spin swirl tilt twist upward vertigo
Detached view
"watching television" "in the distance" "like a movie" "like a television" "watching a movie" "on television" "watch tv" "look out the window" "look out a window" "shown on television" "on the screen"
Negation
neither never no nor not aren't didn't don't doesn't can't hadn't haven't hasn't isn't shouldn't wasn't won't wouldn't

Regular expressions

You can search dreams for regular expression using the "Regex" command (found in the Search panel and the Search menus of dreams and result windows). Regular expressions are strings of characters that define a pattern to be found in a target text. All searches are case sensitive. If you want to make a regular expression case-insensitive prefix it with (?i).

A few examples;

my (mother|father|mom|dad)
meaning: any sequence starting with "my ", followed by one of "mother" or "father" or "mom" or "dad".
matches: "my mother", "my father", "my mom", "my dad"
fl(y|oat){1}(ing|s){0,1}
meaning: any sequence starting with "fl", followed by exactly one of "y" or "oat", followed by zero or one of "ing" or "s".
matches: "fly", "flys", "float", "floats", "flying" and "floating"
[^a-c]at
meaning: any sequence NOT starting with "a", "b" or "c" and ending with "at".
matches: "rat" and "hat" but doesn't match "cat" or "bat"
[mf][oa]ther(.in.law)?
meaning: any sequence starting with "m" or "f", followed by "o" or "a", followed by "ther", followed by an optional ".in.law", where the "." character can be any character.
matches: "mother", "mother-in-law", "father", "father in law" (no hyphens) and even "fatherXinXlaw"
\bdog\b
meaning: the sequence "dog", but only if found on a word boundary.
matches: "dog" but not "doggerel" or "hotdog"
(To match the last two use \bdog\B and \Bdog\b.)

Because regular expression searches can return anything (the whole text of a dream, parts of words, all of the punctuation characters in a dream) there's no simple way to score the dreams for how well they matched the search pattern. (You could count the number of matches returned for each dream and then weight them by the rarity of each match. But this would only work for "sensible" results.) Therefore no scores are given to the individual dreams in clustered versions of Regex reports.

Here are a few useful regular expressions;

metamorphosis
\b((switch|change|turn)s{0,1} (out|into|to))|((bec(a|o)me)s{0,1}\b (a|an|the))\b|\b(now (is|a|an))\b
flying
\b(I|I'm|I was) fl(ew|y|oat){1}(ing){0,1}\b
i do
\b(I|I'm) .{2,34}[.?!]("|')?
he says/she says
\b(she|he) says.{2,51}[.?!]("|')?
repeated word
b(w+) 1b

How relatedness is calculated

Relatedness of dreams is calculated from the number of occurrences of terms in a dream and their overall occurrence in all dreams. The list of a dream's term frequencies is treated as a vector. The cosine of the angle between two dream's vectors is determined and is used as a measure of similarity. Documents with a lot of terms in common "point in the same direction" in term space. The cosine of 0 degrees is 1.0 (perfect agreement). The cosine of 90 degrees is 0.0 (no terms in common). This number is multiplied by 100 to make it consistent with other measures. In practice, a value of greater than 20 indicates a shared "topic" between two dreams.

If you set the threshold to zero then the Term related report will automatically adjust it to the median of the score so that low scoring dreams are excluded.


Cluster searches

Cluster searches take a set of terms and phrases and search for dreams that contain them in higher than average numbers. Rare terms and phrases boost the score of dreams containing them.

If you set the threshold to zero the cluster searches will automatically adjust it to the mean score for the cluster so that low scoring dreams are excluded.

q: cluster quality

q is a measure of cluster quality which shows up in Cluster and Term related search results. q is calculated from the sum of squares of the mean term frequency vector divided the sum of the individual dreams' term vectors:

q = 100 * (||mean||/ sum of ||vi||)

q gives some sense of the "single-mindedness" of a set of dreams. If the set is small the q will be higher. You can get some idea of the quality of your result set by finding the q of a random set of dreams of the same size as yours. (Use the Random Cluster command.)

See the Dhillon and Modha reference for a more detailed explanation of cluster quality.

Anti-Cluster

The anti-cluster report ("~ Cluster") runs a cluster report but then subtracts the cluster from the complete set of dreams. The dreams not found in the cluster are shown in the result. Enter a threshold of "-1" to completely exclude dreams with even a low score in the "positive" cluster.

An interesting result is to find all dreams without characters. To find these dreams I used the following search string in an anti-cluster:

them who his him her we they kid child children baby female she students people somebody someone stranger woman women girl girlfriend lady mom dad male he man men boy boyfriend gentleman guy "group of people" "my mother" "my sister" "my daughter" "my father" "my brother" "my son"
This returns a very small cluster of very short dreams which feature a zoomed-in point of view, static imagery and little emotion.

Best terms

Best terms are those terms that occur in all or most of the dreams in a cluster. Terms that occur in all of the dreams are shown red. Those that occur in greater than half of the dreams (but not all) are shown in orange. Terms that occur in one third to one half of the dreams are shown in blue.

Example of best terms from a cluster of 20 dreams:
| 20: train | 12: station | 11: subwai | 10: platform | 7: line push singl slightli track

Best terms links appear at the bottom of any report that returns a cluster of dreams;

Frequencies

All cluster searches show the raw frequencies of each term in the cluster. This is the sum of the occurrence of a term in all of the cluster's dreams. Only the 40 most frequent terms are shown.

Example frequencies from a cluster of 31 dreams:
| 136: fly | 18: float | 17: beach | 16: fenc | 14: ian | 13: yard ball | 12: pond meet inch | 11: roof | 10: wooden women wet tub tree polic path jetti | 9: tall squar sidewalk moment leg hole hit catch bodi | 8: watch stone push marsh land island hot hair foot dune | 7: test talk

Characteristic terms

Characteristic terms are those terms that score highest in a particular cluster. Unlike best terms, each characteristic term can only occur in a single cluster. Because of this characteristic terms can only be shown in clusters produced by AutoCluster which partitions a set of dream into mutually exclusive sets. Characteristic terms are displayed in a link in purple text. If you click on the characteristic terms they are entered into the cluster's search field so you can do other searches on them.

Example characteristic terms from a cluster of 31 dreams:
fly float yard ball pond inch roof wooden wet tub jetti tall squar sidewalk moment leg hit catch bodi stone

Global and local searches

Cluster searches also display a "Global search" link ([Global search]) for Best Terms, Frequencies and Characteristic Terms. Clicking on this link enters the terms into the Search panel's text field. You can also do the same searches limited to the dreams a cluster: click directly on the terms. For reports with multiple clusters of dreams, like k-means, clicking "Detach dreams" pops up a new window with just the dreams in the selected theme/cluster.

Modules folder

The modules folder allows you to access several stand-alone applications from Bag O' Dreams. Modules must be Java applications in jar format. Modules show up near the end of the second drop-down list of reports on the Search panel

MenuEditor

Allows you to add and edit menu items (lists of words) to the Search panel's menus. A restart of Bag O' Dreams is required before new menu items are shown.

Thesaurus

The thesaurus is a stand-alone application which uses Grady Ward's Moby thesaurus file and a "sounds like" lookup algorithm from William Brogden. It's another java application. For installation and usage see the READ ME file in the distribution. Download it here thesaurus.zip.

If you install the Thesaurus it becomes available on the Dictionary panel.

Set operations

A stand alone module which allows you to do some set operations on two lists of words. Find the intersection, sum and set difference between the lists of words.

RPN Calculator

A reverse-polish notation calculator. You enter arguements first followed by operations. Example: to calculate "log (14*12)" you would key in;
14, ENTER, 12, *, log
instead of;
14, *, 12 ,= , log.

h-Score

Calculate h-score for differences in occurrence of a category between two groups of dreamers. (See Schneider and Domhoff http://psych.ucsc.edu/dreams/Info/statistics.html for explanation and example.)


Settings folder

The settings folder contains files used to store settings and resources used by Bag O' Dreams. Only edit files after you have a good idea what they do. Always make a backup copy of the settings files before you edit them. The following is a list of the files in the settings folder;
anti_stop_words.txt
Force these words to be included in the term table.
clustersettings.txt
The saved settings for the AutoCluster panel.
co_usermenu.txt
The saved settings for the Co-occurrence panel.
interpretations.txt
List of "definitions" shown in the Dictionary panel and in Interpretation reports.
mydreamslocation.txt
Last saved location of the set of dreams.
names.txt
List of names used by the Names report.
stop_words.txt
Exclude this list of words from the term table.
themesettings.txt
The maximum number of words to include in Best Terms and Frequencies links.
menus.txt
All of the menu item on the Search panel.
settings.txt
Settings for the Settings panel.
attributes.txt
Search terms used by the ARFF report.
tfidf.txt
Search terms used by the TFIDF report.
categories.txt
List of word categories used by the Quick Profile report.
regex_categories.txt
List of regular expressions used as categories used by the Quick Profile report.
colors.txt
List of regular expressions used by the Color Profile report.
cloudnine.txt
List of word categories used by the Cloud 9 report.

Tips

How to find all characters, animals, settings, etc.

Bag O' Dreams provides several lists of words useful for identifying dream elements that are commonly of interest to dream analysts. These include: To use the lists with a particular set of dreams follow these instructions:
  1. Enter the set of dreams via the "Add" button. (Usually you'll want to uncheck the option for Porter stemming.)
  2. Select the "Terms from file" command from the Search Panel.
  3. Select one of the files from the list above. (Download it first: see the links above.)
  4. After some time the text field in the Search panel will be filled in with words and phrases from the file that occur in the set of dreams.
  5. Run the "Frequency" report and save the results or copy them into an appropriately named menu item.

Note that items can belong to more than one category: "April" can refer to the month or a woman's name.

"...no book or even computer software could ever list the cast of thousands who have appeared in your dreams..."
-Gayle Delaney, In Your Dreams, p. 163.

The Names report generates a list of possible character names found in a set of dreams. To narrow down the results to people likely to be important to the dreamer, set the threshold field to 1.0 and run a Frequency report on the list of names. Characters who are important to a dreamer usually appear with a frequency greater than 1 per 10000 words. To find out who these characters are run a Pairs report on them after adding phrases such as; "my sister", "my brother", "my daughter", "my son", etc.

How to "disambiguate" closely related themes

Suppose you want find all dreams where you're flying or floating in the air. If you do a Cluster search with 'fly' and 'float' as search terms you'll get a lot of flying dreams, but you might also get a lot of dreams about swimming or floating in water. To separate flying from swimming try a query expansion using the Expand report. In my own dreams an Expand report on "fly float" returns the following search terms;
float tree branch air fly higher surfac lake abil low easili swim altitud ball straight fenc prove
A few of these terms have to do with swimming and a Cluster search returns dreams about flying and swimming. To exclude swimming-related dreams I used "fly float -swim" and got these search terms;
float air fly higher abil low twenti altitud height roof
When I did a Cluster search with the above terms I got much better results. Note that this doesn't exclude all dreams with swimming because I sometimes fly over water or fly and swim in the same dream.

Other Tips

Save your results

Some searches take a long time to complete (e.g. AutoCluster and Interpretation reports). Save these results as HTML documents and reopen them later when you have more time to study them. If you save them in the "settings" folder they will be automatically loaded in the list on the "About" panel the next time you start Bag O' Dreams.

Use the User menu for temporary search lists

Use the MenuEdit utility (in the "modules" folder) to add several temporary menu items to the User menu on the search panel. (Name them "Temp1", "Temp2", "Temp3".) Use these when you are developing new search lists. Don't forget to use the "Save terms" command (also on the Search panel, in the lower left drop-down selector).

Searching for phrases

You can include phrases "words in quotes" in Frequency, Find in files, Terms In Context and Cluster and most other searches.

Large dream sets and memory problems

If your dream set is much larger than 1500 (and depending on your system configuration) you may find that you run out of memory. If this is a problem launch Bag O' Words from a command line with:
java -Xmx128m -jar bag.jar

If you are using Windows, you can save the command line in a bat file (e.g.: "BOD.bat"). You can launch Bag O' Dreams by double-clicking on the bat file. This is especially useful if you find a bug because running from bat file will open a system console where error messages are displayed.

You can also add less than all of the dreams in the set. Use the drop down next to the Add button (on the Settings panel) to select a suitable number of dreams.


© 08/04/2006 by Paul Duffy (duffy.paul@comcast.net).