Introduction to newLISP/Contexts

Introducing contexts

We all like to organize our stuff into separate areas or compartments. Chefs keep their fish, meat, and dessert areas separate, electronics engineers keep their power supplies away from their radio frequency and audio stages, and newLISP programmers use contexts to organize their code.

What is a context?

A newLISP context provides a named container for symbols. Symbols in different contexts can have the same name without clashing. So, for example, in one context I can define the symbol called meaning-of-life to have the value 42, but, in another context, the identically-named symbol could have the value dna-propagation, and, in yet another, worship-of-deity.

Unless you specifically choose to create and/or switch contexts, all your newLISP work is carried out in the default context, called MAIN. So far in this document, when new symbols have been created, they've been added to the MAIN context.

Contexts are very versatile - you can use them for dictionaries, or software objects, or super-functions, depending on the task in hand.

Contexts: the basics

The context function can be used for a number of different tasks:

• to create a new context

• to switch from one context to another

• to retrieve the value of an existing symbol in a context

• to see what context you're in

• to create a new symbol in a context and assign a value to it

newLISP can usually read your mind, and knows what you want to do, depending on how you use the context function. For example:

(context 'Test)

creates a new context called Test, as you might expect. If you type this in interactively, you'll see that newLISP changes the prompt to tell you that you're now working in another context:

> (context 'Test)
Test
Test> 

And you can switch between contexts freely:

> (context MAIN)
MAIN
> (context Test)
Test
Test> 

Used on its own, it just tells you where you are:

> (context)
MAIN
>

Once a context exists, you don't have to quote the name (but you can if you like). Notice that I've used an upper-case letter for my context name. This is not compulsory, just a convention.

A context contains symbols and their values. There are various ways to create a symbol and give it a value.

> (context 'Doyle "villain" "moriarty")
"moriarty"
>

This creates a new context - notice the quote, because newLISP hasn't seen this before - and a new symbol called "villain", with a value of "Moriarty", but stays in the MAIN context. If the context already exists, you can omit the quote:

> (context Doyle "hero" "holmes")
"holmes"
> 

To obtain the value of a symbol, you can do this:

> (context Doyle "hero")
"holmes"
>

or, if you're using the console, this step by step approach:

> (context Doyle)
Doyle
Doyle> hero
"holmes"
Doyle> 

or, from the MAIN context:

> Doyle:hero
"holmes"
>

The full address of a symbol is the context name, followed by a colon (:), followed by the symbol name. Always use the full address if you're in another context.

To see all the symbols inside a context, use symbols to produce a list:

(symbols Doyle)
;-> (Doyle:hero Doyle:period Doyle:villain)

or, if you're already inside the Doyle context:

> (symbols)
;-> (hero period villain)

You can use this list of symbols in the usual way, such as stepping through it with dolist.

(dolist (s (symbols Doyle))
 (println s))

Doyle:hero
Doyle:period
Doyle:villain

To see the values of each symbol, use eval to find its value, and term to return just the symbol's name.

(dolist (s (symbols Doyle))
 (println (term s) " is " (eval s)))

hero is Holmes
period is Victorian
villain is Moriarty

There's a more efficient (slightly faster) technique for looping through symbols in a context. Use the dotree function:

(dotree (s Doyle)
 (println (term s) " is " (eval s)))

hero is Holmes
period is Victorian
villain is Moriarty

Creating contexts implicitly

As well as explicitly creating contexts with context, you can have newLISP create contexts for you automatically. For example:

(define (C:greeting) 
  (println "greetings from context " (context)))
 
(C:greeting)

greetings from context C

Here, newLISP has created a new context C and a function called greeting in that context. You can create symbols this way too:

(define D:greeting "this is the greeting string of context D")
(println D:greeting)

this is the greeting string of context D

In both these examples, notice that you stayed in the MAIN context.

The following code creates a new context L containing a new list called ls that contains strings:

(set 'L:ls '("this" "is" "a" "list" "of" "strings"))
;-> ("this" "is" "a" "list" "of" "strings")

Functions in context

Contexts can contain functions and symbols. To create a function in a context other than MAIN, either do this:

(context Doyle)                         ; switch to existing context
(define (hello-world)                   ; define a local function
 (println "Hello World"))

or do this

(context MAIN)                          ; stay in MAIN
(define (Doyle:hello-world)             ; define function in context
 (println "Hello World"))

This second syntax lets you create both the context and the function inside the context, while remaining safely in the MAIN context all the time.

(define (Moriarty:helloworld)
 (println "(evil laugh) Hello World"))

You don't have to quote the new context name here because we're using define, and define (by definition) isn't expecting the name of an existing symbol.

To use functions while you're in another context, remember to call them using this context:function syntax.

The default function

If a symbol in a context has the same name as the context, it's known as the default function (although in fact it can be either a function, or a symbol containing a list or a string). For example, here is a context called Evens, and it contains a symbol called Evens:

(define Evens:Evens (sequence 0 30 2))
;-> (0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30)
Evens:Evens
;-> (0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30)

And here is a function called Double in a context called Double:

(define (Double:Double x)
   (mul x 2))

So Evens and Double are the default functions for their contexts.

There are lots of good things about default functions. If the default function has the same name as the context, it is evaluated whenever you use the name of the context in expressions, unless newLISP is expecting the name of a context. For example, although you can always switch to the Evens context by using the context function in the usual way:

> (context Evens)
Evens
Evens> (context MAIN)
MAIN
>

you can use Evens as a list (because Evens:Evens is a list):

(reverse Evens)
;-> (30 28 26 24 22 20 18 16 14 12 10 8 6 4 2 0)

You can use the default function without supplying its full address. Similarly, you can use the Double function as an ordinary function without supplying the full colon-separated address:

> (Double 3)
6

You can still switch to the Double context in the usual way:

> (context Double)
Double
Double>

newLISP is smart enough to be able to work out from your code whether to use the default function of a context or the context itself.

Passing parameters by reference

There are important differences between default functions when used as symbols and their more ordinary siblings. When you use a default function to pass data to a function, newLISP uses a reference to the data rather than a copy. For larger lists and strings, references are much quicker for newLISP to pass around between functions, so your code will be faster if you can use store data as a default function and use the context name as a parameter.

Also, and as a consequence, functions change the contents of any default functions passed as reference parameters. Ordinary symbols are copied when passed as parameters. Observe the following code. I'll create two symbols, one of which is a 'default function', the other is a plain symbol:

(define Evens:Evens (sequence 0 30 2))   ; symbol is the default function for the Evens context
;-> (0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30)

(define odds (sequence 1 31 2))          ; ordinary symbol
;-> (1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31)

; this function reverses a list

(define (my-reverse lst)
  (reverse lst))

(my-reverse Evens)                       ; default function as parameter
;-> (30 28 26 24 22 20 18 16 14 12 10 8 6 4 2 0)

(my-reverse odds)                        ; ordinary symbol as parameter
;-> (31 29 27 25 23 21 19 17 15 13 11 9 7 5 3 1)

So far, it looks as if they're behaving identically. But now inspect the original symbols:

> Evens:Evens
(30 28 26 24 22 20 18 16 14 12 10 8 6 4 2 0)
> odds
(1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31)

The list that was passed as a default function - as a reference - was modified, whereas the ordinary list parameter was copied, as usual, and wasn't modified.

Functions with a memory

In the following example, we create a context called Output, and a default function inside it, also called Output. This function prints its arguments, and increments a counter by the number of characters output. Because the default function has the same name as the context, it is executed whenever we use the name of the context in other expressions.

Inside this function, the value of a variable counter (inside the context Output) is incremented if it exists, or created and initialized if it doesn't. Then the function's main task - the printing of the arguments - is done. The counter symbol keeps count of how many characters were output.

(define (Output:Output)                 ; define the default function
 (unless Output:counter
  (set 'Output:counter 0))
 (inc Output:counter (length (string (args))))
 (map print (args))
 (println))

(dotimes (x 90)
 (Output                                ; use context name as a function
 "the square root of " x " is " (sqrt x)))

(Output "you used " Output:counter " characters")

the square root of 0 is 0
the square root of 1 is 1
the square root of 2 is 1.414213562
the square root of 3 is 1.732050808
the square root of 4 is 2
the square root of 5 is 2.236067977
the square root of 6 is 2.449489743
the square root of 7 is 2.645751311
the square root of 8 is 2.828427125
the square root of 9 is 3
...
the square root of 88 is 9.38083152
the square root of 89 is 9.433981132
you used 3895 characters

The Output function effectively remembers how much work it's done since it was first created. It could even append that information to a log file.

Think of the possibilities. You could log the usage of all your functions, and bill users according to how often they use them.

You can override the built-in println function so that it uses this code instead when it's called. See On your own terms.

Dictionaries and tables

A common use for a context is a dictionary: an ordered set of unique key/value pairs, arranged so that you can obtain the current value of a key, or add a new key/value pair. newLISP makes creating dictionaries easy. To illustrate, I'll enlist the help of the great detective, Sherlock Holmes. First, I downloaded Sir Arthur Conan Doyle's The Sign of Four from Project Gutenberg, then I loaded the file as a list of words.

(set 'file "/Users/me/Sherlock Holmes/sign-of-four.txt")
(set 'data (clean empty? (parse (read-file file) "\\W" 0)))  ;read file and remove all white-spaces, returns a list.

Next, I define an empty dictionary:

(define Doyle:Doyle)

This defines the Doyle context and the default function, but leaves that default function uninitialized. If the default function is left empty, you can use the following expressions to build and examine a dictionary:

• (Doyle key value) - set key to value

• (Doyle key) - get value of key

• (Doyle key nil) - delete key

To build a dictionary from the word list, you scan through the words, and, if the word is not in the dictionary, add it as the key, and set the value to 1. But if the word is already in the dictionary, get the value, add 1 to it, and save the new value:

(dolist (word data)
  (set 'lc-word (lower-case word))
  (if (set 'tally (Doyle lc-word))
      (Doyle lc-word (inc tally))
      (Doyle lc-word 1)))

This shorter alternative eliminates the conditional:

(dolist (word data)
   (set 'lc-word (lower-case word))
   (Doyle lc-word (inc (int (Doyle lc-word)))))

or, even shorter still:

(dolist (word data)
   (Doyle (lower-case word) (inc (Doyle (lower-case word)))))

Each word is added to the dictionary, and the value (the number of occurrences) increased by 1. Inside the context, the names of the keys have been prefixed with an underscore ("_"). This is so that nobody gets confused between the names of keys and newLISP reserved words, many of which occur in Conan-Doyle's text.

There are various ways you can browse the dictionary. To look at individual symbols:

(Doyle "baker")
;-> 10
(Doyle "street")
;-> 26

To look at the symbols as they are stored in a context, work through the context evaluating each symbol, using dotree:

(dotree (wd Doyle)
   (println wd { } (eval wd)))

Doyle:Doyle nil
Doyle:_1 1
Doyle:_1857 1
Doyle:_1871 1
Doyle:_1878 2
Doyle:_1882 3
Doyle:_221b 1
...
Doyle:_your 107
Doyle:_yours 7
Doyle:_yourself 9
Doyle:_yourselves 2
Doyle:_youth 3
Doyle:_zigzag 1
Doyle:_zum 2

To see the dictionary as an association list, use the dictionary name on its own. This creates a new association list::

(Doyle)
;-> (("1" 1) 
 ("1857" 1) 
 ("1871" 1) 
 ("1878" 2) 
 ("1882" 3) 
 ("221b" 1) 
 ...
 ("you" 543) 
 ("young" 19) 
 ("your" 107) 
 ("yours" 7) 
 ("yourself" 9) 
 ("yourselves" 2) 
 ("youth" 3) 
 ("zigzag" 1) 
 ("zum" 2))

This is a standard association list, which you can access using the functions described in the Lists chapter (see Association lists). For example, to find all words that occur 20 times, use find-all:

(find-all '(? 20) (Doyle) (println $0))
;-> ("friends" 20)
("gone" 20)
("seemed" 20)
("those" 20)
("turned" 20)
("went" 20)

The association list returned by (Doyle) is a temporary copy of the data in the dictionary, not the original dictionary context. To change the data, don't operate on this temporary list but on the context's data, using the key/value access techniques.

You can also add new entries to dictionaries, or modify existing entries, using data in the form of an association list:

(Doyle '(("laser" 0) ("radar" 0)))

Saving and loading contexts

If you want to use the dictionary again, you can save the context in a file:

(save "/Users/me/Sherlock Holmes/doyle-context.lsp" 'Doyle)

This collection of data, wrapped up in a context called Doyle, can be quickly loaded by another script or newLISP session using:

(load "/Users/me/Sherlock Holmes/doyle-context.lsp")

and newLISP will automatically recreate all the symbols in the Doyle context, switching back to the MAIN (default) context when done.

Using newLISP modules

Contexts are used as containers for software modules because they provide lexically-separated namespaces. The modules supplied with the newLISP installation usually define a context that contains a set of functions handling tasks in a specific area.

Here's an example. The POP3 module lets you check POP3 email accounts. You first load the module:

(load "/usr/share/newlisp/modules/pop3.lsp")

The module has now been added to the newLISP system. You can switch to the context:

(context POP3)

and call the functions in the context. For example, to check your email, use the get-mail-status function, supplying user name, password, and POP3 server name:

(get-mail-status "someone@example.com" "secret" "mail.example.com")
;-> (3 197465 37)
; (totalMessages, totalBytes, lastRead)

If you don't switch to the context, you can still call the same function by supplying the full address:

(POP3:get-mail-status "someone@example.com" "secret" "mail.example.com")

Scoping

You've already seen the way newLISP dynamically finds the current version of a symbol (see Scope). However, when you use contexts, you can employ a different approach, which programmers call lexical scoping. With lexical scoping, you can explicitly control which symbol is used, rather than rely on newLISP to keep track of similarly-named symbols for you automatically.

In the following code, the width symbol is defined inside the Right-just context.

(context 'Right-just) 
(set 'width 30)
(define (Right-just:Right-just str)
  (slice (string (dup " " width) str) (* width -1)))

(context MAIN)
(set 'width 0)                         ; this is a red herring
(dolist (w (symbols))
  (println (Right-just w)))

                             !
                            !=
                             $
                            $0
                            $1
                           ...
                    write-line
                     xml-error
                     xml-parse
                 xml-type-tags
                         zero?
                             |
                             ~

The second (set 'width ...) line is a red herring: changing this here makes no difference at all, because the symbol which is actually used by the right-justification function is inside a different context.

You can still reach inside the Right-just context to set the width:

(set 'Right-just:width 15)

There's been much discussion about the benefits and disadvantages of the two approaches. Whatever you choose, make sure you know where the symbols are going to get their values from when the code runs. For example:

(define (f y) 
  (+ y x))

Here, y is the first argument to the function, and is independent of any other y. But what about x? Is it a global symbol, or has the value been defined in some other function that has just called f? Or perhaps it has no value at all!

It's best to avoid using these free symbols, and to use local variables (defined with let or local) wherever possible. Perhaps you can adopt a convention such as putting asterisks around a global symbol.

Objects

More has been written about object-oriented programming (OOP) than you could possibly read in one lifetime, so this section is just a quick glance at the subject. newLISP is agile enough to enable more than one style of OOP, and you can easily find references to these on the web, together with discussions about the merits of each.

For this introduction, I'll briefly outline just one of these styles: FOOP, or Functional Object-Oriented Programming.

FOOP in a nutshell

FOOP has changed with newLISP version 10.2 (beginning of 2010), so if you're using and old version of newLISP, please update it.

In FOOP, each object is stored as a list. Class methods and class properties (ie functions and symbols that apply to every object of that class) are stored in a context.

Objects are stored in lists because lists are fundamental to newLISP. The first item in an object list is a symbol identifying the class of the object; the remaining items are the values that describe the properties of an object.

All the objects in a class share the same properties but those properties can have different values. The class can also have properties that are shared between all objects in the class; these are the class properties. Functions stored in the class context provide the various methods for managing the objects and processing the data they hold.

To illustrate these ideas, consider the following code that works with times and dates. It builds on top of the basic date and time functions provided by newLISP (see Working with dates and times). A moment in time is represented as a time object. An object holds two values: the number of seconds that have elapsed since the beginning of 1970, and the time zone offset, in minutes west of Greenwich. So the list to represent a typical time object looks like this:

(Time 1219568914 0)

where Time is a symbol representing the class name, and the two numbers are the values of this particular time (these numbers represent a time around 10 in the morning on Sunday August 24 2008, somewhere in England).

The code required to build this object is straightforward using newLISP generic FOOP constructor:

(new Class 'Time) ; defines Time context

(setq some-england-date (Time 1219568914 0))

However, you might want to define a different constructor, for example, you might want to give this object some default values. To do that you have to redefine the default function which is acting as the constructor:

(define (Time:Time (t (date-value)) (zone 0))
    (list Time t zone))

It's a default function for the Time context that builds a list with the class name in the first position, and two more integers to represent the time. When values are not supplied, they default to the current time and zero offset. You can now use the constructor without supplying some or any parameters:

(set 'time-now (Time))
;-> your output *will* differ for this one but will be something like (Time 1324034009 0)
(set 'my-birthday (Time (date-value 2008 5 26)))
;-> (Time 1211760000 0)
(set 'christmas-day (Time (date-value 2008 12 25)))
;-> (Time 1230163200 0)

Next, you can define other functions to inspect and manage time objects. All these functions live together in the context. They can extract the seconds and zone information by picking them from the object by using the self function. So (self 1) gets the seconds and (self 2) gets the time zone offset from the object passed as parameter. Notice the definitions do not require you to state the object parameter. Here are a few obvious class functions:

(define (Time:show)
   (date (self 1) (self 2)))

(define (Time:days-between other)
   "Return difference in days between two times."
   (div (abs (- (self 1) (other 1))) (* 24 60 60)))

(define (Time:get-hours)
   "Return hours."
   (int (date (self 1) (self 2) {%H})))

(define (Time:get-day)
   "Return day of week."
   (date (self 1) (self 2) {%A}))

(define (Time:leap-year?)
   (let ((year (int (date (self 1) (self 2) {%Y}))))
      (and (= 0 (% year 4)) 
         (or (!= 0 (% year 100)) (= 0 (% year 400))))))

These functions are called by using the colon operator and providing the object which we want the function to act upon:

; notice 'show' uses 'date' which works with local time, so your output probably will differ
(:show christmas-day)
;-> Thu Dec 25 00:00:00 2008
(:show my-birthday)
;-> Mon May 26 01:00:00 2008

Notice how we used the colon operator as a prefix to the function, without separating it with a space, this is a matter of style, you can use it with ot without spaces:

(:show christmas-day) ; same as before
;-> Thu Dec 25 00:00:00 2008
(: show christmas-day) ; notice the space between colon and function
;-> Thu Dec 25 00:00:00 2008

This technique allows newLISP to provide a feature that object-oriented programmers love: polymorphism.

Polymorphism

Let's add another class which works on durations - the gap between two time objects - measured in days.

(define (Duration:Duration (d 0))
   (list Duration d))

(define (Duration:show)
   (string (self 1) " days "))

There's a new class constructor Duration:Duration for making new duration objects, and a simple show function. They can be used with the time objects like this:

; define two times
(set 'time-now (Time) 'christmas-day (Time (date-value 2008 12 25)))

; show days between them using the Time:days-between function

(:show (Duration (:days-between time-now christmas-day)))
;-> "122.1331713 days "

Compare that :show function call with the :show in the previous section:

(:show christmas-day)
;-> Thu Dec 25 00:00:00 2008

You can see that newLISP is choosing which version of the show function to evaluate, according to the class of :show's parameter. Because christmas-day is a Time object, newLISP evaluates Time:show. But when the argument is a Duration object, it evaluates Duration:show. The idea is that you can use a function on various types of object: you perhaps don't need to know what class of object you're dealing with. With this polymorphism, you can apply the show function to a list of objects of differing types, and newLISP selects the appropriate one each time:

(map (curry :show) 
    (list my-birthday (Duration (:days-between time-now christmas-day))))

;-> ("Mon May 26 01:00:00 2008" "123.1266898 days ")

Note: We have to use curry here because map needs to use both the colon operator as well as the show function.

Modifying objects

We're calling this particular style of OOP FOOP because it's considered to be functional. Here, the term 'functional' refers to the style of programming encouraged by newLISP which emphasizes the evaluation of functions, avoiding state and mutable data. As you've seen, many newLISP functions return copies of lists rather than modify the originals. There are, though, a small number of functions that are called destructive, and these are considered to be less purely functional. But FOOP doesn't provide for destructive object methods, so can be considered more functional.

A key point to notice about FOOP is that objects are immutable; they can't be modified by class functions. For example, here's a function for the Time class that adds a given number of days to a time object:

(define (Time:adjust-days number-of-days)
  (list Time (+ (* 24 60 60 number-of-days) (self 1)) (self 2)))

When this is called, it returns a modified copy of the object; the original is unchanged:

(set 'christmas-day (Time (date-value 2008 12 25)))
;-> (Time 1230163200 0)

(:show christmas-day)
;-> "Thu Dec 25 00:00:00 2008"

(:show (:adjust-days christmas-day 3))
;-> "Sun Dec 28 00:00:00 2008"

(:show christmas-day)
;-> "Thu Dec 25 00:00:00 2008"
; notice it's unchanged

The original date of the christmas-day object didn't change, although the :adjust-days function returned a modified copy adjusted by 3 days.

In other words, to make changes to objects, use the familiar newLISP approach of using the value returned by a function:

(set 'christmas-day (:adjust-days christmas-day 3))

(:show christmas-day)
;-> "Sun Dec 28 00:00:00 2008"

christmas-day now contains the modified date.

You can find a more complete working out of this idea by searching the newLISP forum for timeutilities. Also make sure to read the section on FOOP in the reference manual, it has a nice example on nested objects, that is, objects containing other objects.