Io module system discussions
Introduction
This document is related to the ModuleSystem proposal. It contains the discussion we had regarding the conception of the module system. As the system is not finished, this document is still living actively, so you are invited to continue the discussion :)
Related pages:
Discussion
Scott's conjecture
The following is just a conjecture by sdunlop:
Creating a module:
MyModule = Module clone("MyModule") document ("""
This module implements ...""")
MyModule load = method(
self MyPrototype? = Object clone documentPrototype ( ... )
MyPrototype init = method( ... ) document ( ... )
)
MyModule setup = method( ... )
MyModule teardown = method( ... )
MyModule author = "Scott Dunlop"
MyModule author-email = "sdunlop@midcoast.com"
MyModule author-website = "http://www.midcoast.com/~sdunlop"
MyModule majorVersion = 0
MyModule minorVersion = 1
MyModule stability = ALPHA
export(MyModule)
Importing a Module and a Prototype:
HisModule = importModule("MyModule")
HisPrototype = HisModule import ("MyPrototype")
Alternatively, the classic naive "from module import *" equivalent:
importAllFromModule("HisModule")
Sebastien's alternative
Here is an alternative proposal, using a IoPackage?? object (I make a difference between packages and modules - think of packages as nodes and modules as leaves):
#Import all from a module
IoPackage from ("MyModule") importAll
#Import protos from a module into the Lobby
IoPackage from ("MyModule") import ("MyProto")
IoPackage from ("MyPackage1.MyPackage2.MyModule") import ("Proto1", "Proto2")
#Import protos from a module into the Lobby as another Proto
IoPackage from ("MyModule") import ("MyProto") as ("RenamedProto")
IoPackage from ("MyModule") import ("MyProto", "MyProto2") as ("RenamedProto1","RenamedProto2")
#Import only a module/package into the lobby
IoPackage import ("MyModule")
#Import only a module/package and rename it
IoPackage import ("MyModule") as ("RenamedModule")
IoPackage? would support resolution of modules given the dot-separated module name. Forinstance "Package1.Module1" would first look for the "Package1" somewhere (localdirectories, tarballs, URLs?) and then when Package1 is found, ask resolution of Module1to Package1.
I think it is important to provide a IoPackage? object because we could implement different strategies for importing modules, some that may take into account security, orprovide fancy loading features.
Scott's addition
I prefer the verbosity of your method to my own, but only slot names composed entirely of symbols can be used as operators in Io, therefore, something like:
IoPackage import "MyModule" as "RenamedModule"
should be expressed as:
IoPackage import("MyModule") as ("RenamedModule")
Cool :) I fixed this --sebastien
Daniel on interacting modules
The major issue, for me, is how to keep the modules from interacting with each other in unconstrained ways. Consider three modules, main.io:
require("Foo")
require("Bar")
Foo.io:
foobar = 42
and Bar.io:
write(Foo foobar, "\n")
Unless this is explicitly forbidden somehow, these modules are legal but ill-formed. Bar depends upon the existence of Foo, but it does not state this requirement. Thus our dependency graph is incomplete and the meaning of Bar changes based upon its context. It might raise an error because "Foo" or "Foo foobar" don't exist, print 42, or do something completely different. You can't really tell what Bar is going to do by looking at its code.
How then do we restrict access between two modules within the current framework of Io? The only way I can think of doing it is by changing the meaning of Lobby. Instead of Lobby being the object in which all global code is run, Lobby becomes a system-defined module which is implicitly imported into all other modules. Every module is its own namespace.
Now consider the above example of main, Foo, and Bar again. The 'main' module requires Foo and Bar, which have not yet been loaded. The system tracks down the implementations associated with Foo and Bar and executes them. Foo runs in its own namespace and then added to main's Protos object (to indicate that main has imported it and can now access it). Now Bar runs in its own namespace, which doesn't define Foo. It looks at its proto, which points to the Protos object, which only contains Lobby -- Lobby also does not define Foo. Thus, we have an error, as one might expect from looking at Bar's code. If Bar is changed to:
require(Foo) write(Foo foobar, "\n")
then the system recognized that the Foo module is already loaded and just adds the existing Foo object to Bar Protos, allowing Bar to access Foo.
Thus, we have the case that if Bar attempts to use Foo's interface, it must either explicitly require Foo or raise an error. This only leaves one hole, whereby modules can interact without explicit requirements between them. If Foo were changed to add foobar to the Lobby instead of its namespace (thisModule), then Bar could access foobar through the Lobby. I think this is unlikely to pose a problem and it may actually be a good thing to allow users to circumvent some of the module system's requirements when they really want to.
Even though this is a significant change to how namespaces are handled in Io, it will only break programs that attempt to refer to the slots they're adding through the Lobby. Those programs would need to be changed to refer use thisModule, instead.
As a last note, since this is already a very long addition to the page (sorry!), I've basically removed the notion of explicitly declaring that a file is a module, in favor of having Io automatically treat the first file it loads and every required file after that implicitly as a module. Having a file declare itself a module breaks doFile() and requires the interpreter to keep track of more information.
I call the module loading facility require() instead of import() because import(), to me, implies that you're making the definitions of a module available inside the current one. I view import() as a separate mechanism:
require("Foo") # Make the Foo module object available.
Foo import("foobar") # Bring in the foobar object.
It may be worthwhile to provide another module loader, request(), which defines the module object as Nil if it fails, instead of throwing an exception (as I imagine require() doing). This might be useful in combination with module versioning:
request("GTK", majorVersion == 2)
if(GTK == Nil) then(
request("GTK", majorVersion < 2)
# Build a compatability layer for older versions of GTK.
)
# code common to all versions of GTK we support.
or in general:
request("ReadLine")
require("File") import("standardInput")
...
# Use full line-editing capabilities, if ReadLine package is available.
expr += if(ReadLine, ReadLine readLine, standardInput readLine)
...
-dak
A comment on request: why not leave it out and rely on import throwing an exception? The reasoning is similar to why it's better to have a file open operation that fails when the file doesn't exist, rather than a seperate "does the file exist" query you're supposed to call beforehand.
--quinn
Sebastien's comments
First, here are some comments on request/import. I like the idea of a request operation with modules, but I think the usage of request and require is too similar, the difference being that request takes attributes and assign Nil to the module instead of throwing an exception.
I would rather prefer a usage of request as a predicate that would tell wether the given module is available with the given attributes/constraints. For instance:
if ( request("Console.ReadLine", version="1.2+"),
"Readline is available" print ; import("Console.ReadLine") as "rl",
"Realline is unavailable"
)
You note that here, the request does not load or initiate but rather answers queries/requests made to the module system. This clearly separates request, which answers questions about modules availability, and require, which actually load them.
However, there is something that bugs me with the require:
require("Foo") as("Pouet")
seems strange, but I found the following clearer
import("Foo") as("Pouet")
Now, I undestand that there is a difference between loading a module and importing a module symbols into the current namespace. What I would propose is to have only import, but use it this way:
import("MyModule") as ("AModule")
AModule import("AnObject", "AnotherObject")
which would be equivalent to
from("MyModule") as("AModule") import("AnObject", "AnotherObject")
But we could also do
from("MyModule") import("AnObject", "AnotherObject")
Note that this would imply that successive imports returns the same module instance (import does not instanciate a module).
Regarding the problem of interacting modules, and the solution you propose of considering the Lobby as a module itself, and files as modules, I must say that I really like your idea :) I also think that there is a very minimal impact on Io, which is important.
It's clear that modules should be insulated from each other, and in some cases the modules should prevent access from other modules (see the Security requirements, in the requirements section). What you propose is a good starting point for such requirements.
However, what you propose raises the question of the semantics of the import operation:
- Does
importimmediatly load the module or is the module only loaded and initialised when referenced ?
[ I suggest it loads immediately, so initialization is done at an obvious time, rather than at some unpredictable point in the future. What if there's an error? Better to catch exceptions at import. If a module wants to initialize lazily, it shouldn't be hard to implement that. ]
- Is there only one module instance per VM (are modules shared, see Design Questions section).
--Sebastien, 19 june 2003
Hung Jung's comments
I am brand new to Io. I come from Python. I'd like to comment on problems of modules in Python. To me, module is just a name space. Module import statements are pretty much like inheritance. In fact, there is little difference between modules and classes/objects. I think Python made a mistake by treating modules differently from name spaces (classes/objects). If Io's goal is unification, I think it's better to think twice and see how modules can be unified with objects. I say this because in Python later people start to introduce "properties" (or better known as getters/setters), so all of a sudden you can intercept accesses to object attributes. So far so good. But then you realize that you can't do the same with modules, since at fundamental level modules are different from classes. I mean, modules are very similar to mundane objects, why not unify them, so that in the future, enhancements you do to one will automatically be applied to the other? Why repeat the mistake of other languages? --Hung Jung, 11 december 2003
Einar Karttunen's comments
Making modules objects sounds very good. In general why should a module be any different from an object (or a hash table). Also why separate packages? One could model them just as modules containing other modules.
In my opinion modules could be regular io source files. When importing a module it would be "evaluated" (that is if it is not precompiled) in a sandbox without permission to alter ""anything"" outside the module. Instead of lobby an empty object would be used in which the "global variables" would be put.
author = "Einar Karttunen" name = "Foo" version = "1.3" MyFoo = Object clone MyFoo bar = method(Nop)
and so on. Of course all the metainformation could be put into an object (or hash) names e.g. "meta".
Importing would work like this:
MyFooModule = import("foo") # make somekind of clever lookup like in Ruby
if(MyFooModule, bring_members_into_scope(MyFooModule), error...)
The idea is that the module would be a first class object. We could either use it's services directly (MyFooModule? MyFoo?) or import them into local scope
which could be done like:
foreach(n, MyFooModule slotNames, setSlot(n, MyFooModule getSlot(n)))
All this would of course be hidden away with a nice API.
--Einar Karttunen, 19 december 2003
WardCunningham's comments
I think Einar is on to something but that it can be even simpler. How about something as simple as:
MyFooAndBarModuleLobby := Lobby clone do (
doFile("Foo.io")
doFile("Bar.io")
At this point Foo.io and Bar.io share a Lobby that does not interfere with me. However, they have access to everything I put in my Lobby (unless I protect myself by making my own private Lobby too.) I can also use MyFooAndBarModuleLobby? to find resources that Foo.io and Bar.io might leave for me. Of course library management will require many conventions built upon this mechanism, like dependency management and unit tests.
Aside: If you are in the habit of adding behavior to system objects like String and Object, then you might prefer that cloned lobbies have cloned copies of these objects too. That means that the behavior of a String might depend on where it came from and that Io itself might have to be sure to access the right version when there are multiple lobbies in one vm.
Sébastien, comments (2004-12-27):
Hi Ward ! It's a pleasure to have you here, you're a start ;)
Now, regarding module, I would not recommend to override existing primitive operations -- as it would become complicated to read code which behaviour depends on the type of loaded modules.
However, the idea of cloning lobbies is interesting !