-*- mode: muse -*- * 2008-02-03, 10:45:58 CET ** To be compatible or not to be compatible? I wonder whether being API-compatible with Clozure CL's Objective-C bridge would be a good or bad thing. On the one hand, compatibility means application portabilitiy, which is nice. On the other hand, using the same package names and reader macros as Clozure CL's bridge makes it hard to have both loaded at the same time. Then again, if someone wants to compare Objective-CL with Clozure CL's bridge side-by-side, it's their responsibility to rename packages as needed. Reader macros aren't essential for using Objective-CL, so they may be left disabled in such a case, anyway. I'm going to try hard to use Objective-CL-specific package names within my code (i.e. OBJECTIVE-C-CLASSES rather than NS), so renaming packages won't break things. I'll be opting for direct API compatibility for now. It seems to be the right choice. * 2008-01-29, 21:34:16 CET In the Objective-C 2.0 runtime, the functions class_add{Method,Protocol,Ivar}, class_copyMethodList, {class,protocol}_copyProtocolList, protocol_copyMethodDescriptionList, and class_copyPropertyList are probably our friends. class_copyMethodList may be used to together with method_setImplementation for good effect. But... What about the GNU runtime? Is it okay to inspect a Class' `methods' member (see objc.h and objc-api.h) and change the IMPs that the individual members point to? Is it possible to add new methods at runtime by using class_add_method_list and thus ObjcUtilities_register_method_list? Is the behaviour of these two functions specified if a method list for a given class has already been registered in the past? If so, do they replace the original list or amend it? Changing ivars after class creation seems generally impossible. This applies to all the supported runtimes, and I think I can actually see why. I don't think it's a serious problem, but I do consider it regrettable. * 2008-01-28, 20:45:35 CET I've added the following files as a first step to support class definition. For the GNU runtime (from JIGS, the Java Interface to GNUstep): * JIGS/ObjcRuntimeUtilities.c * JIGS/ObjcRuntimeUtilities.h * JIGS/ObjcRuntimeUtilities2.m For the NeXT runtime (from PyObjC): * PyObjC/pyobjc-compat.h * PyObjC/objc-runtime-compat.h * PyObjC/objc-runtime-compat.m Both the JIGS and PyObjC codebases are impressively modular. Those guys know what they're doing. * 2008-01-27, 12:55:06 CET Objective-CL now includes its own version of libffi, imported from an older version of PyObjC. (The current version only supports x86 and PowerPC.) Interestingly, according to the Web, Mac OS X 10.5 includes its own version of libffi. This is really good news! We can finally rest assured that Objective-CL does not break on Mac OS X because of libffi bitrot anytime soon. In order to take advantage of this new state of the world, the build system has been changed so as to only compile our own version of libffi if we can't find any on the system. The downside? We now depend on autoconf. I don't consider this a problem, though. * 2007-10-10, 12:02:38 CEST I've cleaned the Objective-C code up by making the NeXT and GNU runtime-specific code converge a bit. This also makes FIND-SELECTOR return NIL for unknown selectors on the NeXT runtime, so compile-time warnings about unknown methods are possible there now. The latter relies on sel_isMapped, whose semantics are not entirely clear to me. On the one hand, Apple's reference manual states: “You can use this function to determine whether a given address is a valid selector,” which I interpret as meaning that it takes a selector pointer as an argument, not a string. On the other hand, in the preceding section, the same document states: “You can still use the sel_isMapped function to determine whether a method name is mapped to a selector.” So if I have two strings that aren't the same under POINTER-EQ, but that both name the same valid selector that is registered with the runtime, like "self", say, does sel_isMapped work reliably in this case? I'm not sure. On another note, I wonder what the difference between sel_get_uid/sel_getUid and sel_register_name/sel_registerName might be. They seem to do the same thing. Maybe this whole #ifdef mess isn't even strictly necessary, anyway. I could just copy objc-gnu2next.h from the GNUstep project (LGPLv3, so the licensing is fine). http://svn.gna.org/svn/gnustep/libs/base/trunk/Headers/Additions/GNUstepBase/objc-gnu2next.h * 2007-10-04, 17:27:02 CEST ** `char' Does Actually Indicate a Char, Sometimes The latest changes made the test cases fail on GNUstep/x86, which either means that the PyObjC code is wrong, or the GNU runtime has very weird calling conventions that use ints as wrappers for chars or something. Anyway, I have reverted the changes for GNUstep and left them in place for Mac OS X (but note that I left the PyObjC code as it is, which means that libffi is still directed to treats chars as ints). As a result, both NeXT/PowerPC and GNUstep/x86 work for now, but I'm uncertain about the status of other architectures as well as calling methods with chars and shorts as arguments, which I've got no test cases for. I'm not confident that either GNUstep/PowerPC/SPARC/whatever or NeXT/x86 work the way my code expects them to. * 2007-10-04, 16:52:32 CEST ** `char' Does Not Indicate a Char, Continued There's a good chance that I've figured out what to do about the char/int mess. As it turns out, it isn't even limited to chars, as shorts are affected, too. According to the code I took from PyObjC, specifically the typespec conversion functions in libffi_support.m, both GNUstep and NeXT/PowerPC treat chars and shorts as ints. The only platform that isn't brain-damaged in this way seems to be NeXT/x86. Or maybe it's even more brain-damaged, as it treats shorts and chars normally when they are used as arguments, but as ints when they're used as return values! At least GNUstep and NeXT/PowerPC are brain-damaged in a *consistent* manner. I figure the reason I never saw this problem in GNUstep is probably endianness. The little-endian x86 lets you treat pointers to ints as pointers to chars without breaking anything, but that doesn't work in big-endian machines. * 2007-10-04, 13:02:31 CEST ** `char' Does Not Indicate a Char In principle, the typespec "c" is supposed indicate a char. Now look at the following SLIME session transcript (SBCL/PowerPC on Mac OS X): OBJECTIVE-CL> (defparameter *tmp* (invoke (find-objc-class 'ns-string) :string-with-u-t-f-8-string "Mulk.")) *TMP* OBJECTIVE-CL> (defparameter *tmp2* (invoke (find-objc-class 'ns-string) :string-with-u-t-f-8-string "Mulk.")) *TMP2* OBJECTIVE-CL> (second ;return type specifier (multiple-value-list (retrieve-method-signature-info (find-objc-class 'ns-string) (selector :is-equal)))) "c" OBJECTIVE-CL> (invoke *tmp* :is-equal *tmp2*) 0 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :char *tmp2*) 0 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :int *tmp2*) 1 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :long *tmp2*) 1 OBJECTIVE-CL> (primitive-invoke *tmp* :is-equal :long-long *tmp2*) 4294967296 Now, I see why the last value is bogus (I'd be surprised if it weren't, actually), but why the heck is the correct value (1, because, you see, the strings *are* equal and +YES+ is 1 on my machine) returned only for the wrong return type? The return type is specified as `c', but it's actually an int! What's going on here? And rather more importantly: What can I do about this? I don't feel exactly comfortable about cheating and treating `c' as specifying an int on all systems based on the NeXT runtime without having any indication about what else there is in the NeXT runtime that has to be special-cased. I haven't seen this weird behaviour documented anywhere. Even this specific case is non-trivial, for I don't know whether this applies to all chars, or only to chars that are booleans, or only to chars that are returned, or even only to chars that are returned *and* are actually booleans. * 2007-09-26, 00:13:11 CEST ** Licensing Licensing is another open question. For the moment, I'm releasing this project under the terms of the GPLv3. This seems like a reasonable choice, because it gives me the option of giving people more permissions later by applying the LGPLv3 to my code. I must be aware that only I am allowed to do this, though, and even then only if all contributors agree (if someone actually makes a contribution, that is). I may want to require all contributors to dual-license their contributions, or maybe to make them available under the terms of the LGPLv3 in the first place (though the latter would make marking them difficult). * 2007-09-25, 20:59:40 CEST ** Value Conversion Madness Open question: Should NSArray instances be converted to lists or arrays automatically? If so, we ought to make functions like OBJC-CLASS-OF behave in a reasonable way for those kinds of objects, i.e. return NSArray or NSMutableArray (whatever it is that INVOKE makes out of them when converting them into Objective-C instances again). Note that we *must not* convert NSMutableArray instances or any other mutable objects in this way! Note also that our decision *must* be based on the dynamic type of the object, not the static one, because a method whose return type is NSArray may as well return an NSMutableArray that we've fed it sometime earlier. This is okay for immutable objects, but mutable objects are bound to cause trouble when such a thing happens. Related types of objects are strings (NSString), hash tables (NSDictionary), and numbers (NSNumber). Note that such behaviour would make it impossible to fully identify CLOS classes with Objective-C classes, as arrays would have no Objective-C class to belong to. Then again, why would you want to distinguish Objective-C arrays from Lisp arrays in your Lisp code, anyway? Real integration means not having to worry about such things. On the other hand, conversion of large NSArrays may be prohibitively expensive, so a switch is needed, either way. The real question is what the default behaviour should look like. There's an alternative to consider, too. For NSArrays, there is Christophe Rhodes' user-extensible sequence proposal, but even without support for that, we can provide a *conduit* (a package) that looks like the COMMON-LISP package, but overloads all sequence and hash-table functions. Overloading all sequence functions might be a lot of work, though. * 2007-09-23, 17:09:07 CEST ** Improved Memory Management for the Masses Up until now, the second-generation method invocation procedures (LOW-LEVEL-INVOKE and PRIMITIVE-INVOKE) simply called MAKE-INSTANCE for every object received from Objective-C, which meant that although a lookup in the caching hash tables was done, method dispatch for MAKE-INSTANCE was needed. Therefore, everything just worked, but did so slowly. I realised yesterday, after having profiled the code and detected that MAKE-INSTANCE method dispatch was the speed bottleneck of INVOKE calls now, that overriding MAKE-INSTANCE wasn't really necessary for memory management, as we could put instances into the hash tables and register finalisers for them just after they were fully created. So that's what I made the program do. One of the results is much shorter and clearer code, but the more interesting one is a speed improvement of around the factor 3, making 100'000 calls to NSMethodSignature#getArgumentTypeAtIndex:, which previously called MAKE-INSTANCE for each returned value, take around 10s on my machine. With the CFFI speed hack enabled, caching CFFI::PARSE-TYPE results, this figure even goes down to around 2s (that's 50'000 method calls per second). I think that's pretty cool. I'm quite satisfied with method invocation performance now. Compared to C, We're still off by a factor of 22 or so (0.9s for 1'000'000 method calls). Most of the time is spent on memory allocation for argument passing and typespec strings. By introducing a global pool of preallocated memory spaces for these purposes (one argument space per thread and maybe a bunch of string buffers, with a fallback mechanism for method calls that take too much space), we might be able to cut the run time by another factor of 5. After that, we can't optimise the Lisp code any further, because most of the rest of the time is spent within the Objective-C function objcl_invoke_with_types (or maybe in calling it via CFFI, which would be even worse, optimisationwise). It's probably best not to spend too much time pondering this, though, because without the CFFI speed hack, the improvement would probably not be noticeable, anyway (CFFI::PARSE-TYPE is most often called by CFFI:MEM-REF and CFFI:MEM-AREF, not by the allocation routines). ** Milestones Lying Ahead There are three things left to do that are showstoppers against actually using Objective-CL productively. One is support for structs. This one is actually quite a bit harder than it looks, because we don't necessarily know the structure of foreign objects. Objective-C tells us about the structure (though not the member naming!) of structures as well as pointers to structures that are returned by methods, but any more indirection (that is, pointers to pointers to structs or something even hairier) makes the Objective-C runtime conceal the internals of the structs pointed to. This is probably not a problem in practise, though, as pointers to pointers to structs will usually mean a pointer that the user may alter in order to point to other structs, not that the user will access the structs that are pointed to. In fact, it will probably be best to just pass pointers on to the user. The second thing left to do is support for defining Objective-C classes. I think this is going to be hard. I've not looked at the problem in detail yet, but it looks like creating methods and classes, and registering methods and classes are all different actions that are all handled differently depending on the runtime. In the case of GNUstep, I don't even know how to register new selectors yet. Third, varargs. These are easy to implement, but I'm not sure how they should look like in the case of INVOKE. Maybe a special keyword indicator like :* would work for indicating the end of the method name, but I think that could be a bit ugly. We shall see. ** OpenMCL and Objective-CL Compared On another note, I briefly checked out OpenMCL's support for Objective-C by randomly typing a bunch of method invocations into the listener and calling APROPOS a lot. Here's what stuck: 1. You have to explicitely create selectors by using @SELECTOR. Why is that? What's wrong with symbols and strings? 2. Strings designate only NSString objects, not C strings. Why? 3. The bridge is just as fast as I expect using libffi from C to be on that machine, that is, more than 20 times as fast as Objective-CL with the speed hack enabled (250'000 method calls per second; my Inspiron is faster, so don't compare this value to the ones above). 4. There's no FIND-OBJC-CLASS, but FIND-CLASS works. Objective-C classes seem to be normal CLOS classes whose names are found in the NS package. All in all, what struck me the most was the fact that the OpenMCL Objective-C bridge does not seem to make use of the concept of designators as much as Objective-CL does. You have to define C strings and selectors explicitely, which I consider a minor annoyance. It's faster, though. Then again, considering that it's integrated into the compiler, I was bit disappointed by the speed, because I figured that a native-code compiler could do better than libffi (which is still a lot slower than directly calling stuff from Objective-C). ** By The Way Gorm rules. We need to make Objective-CL fully Gorm-compatible.