Digital Nets and Intelligent Systems

K.N.Gurney and M.J.Wright


Department of Human Sciences,
Brunel University,
Uxbridge,
Middx UB8 3PH,
United Kingdom

The journal's aim is to provide research and review papers on an interdisciplinary level, where the focal point is the field of intelligent systems. Neural networks are assemblies of simple processing elements whose connectivity and function are supposed to be, at least loosely, based on that of neurons in the animal brain. Their study can be approached from several different points of view, leading to involvement by workers in many fields and thereby crossing many discipline boundaries. Without wishing to force individual investigators into a particular mould, we can perhaps identify three main themes.

  1. Neuroscientists and psychologists are interested in nets as computational models of the animal brain developed by abstracting, what are believed to be, those properties of real nervous tissue that are essential for information processing.
  2. On the other hand, engineers and computer scientists see neural nets as one style of parallel distributed computing which may usefully be recruited for solving complex problems in pattern recognition and classification, associative memory, and function optimisation. The hope is that they may be more successful in dealing with real-world situations than the conventional algorithmic techniques that have predominated in machine intelligence.
  3. Physicists and mathematicians are drawn to the study of networks from an interest in non-linear dynamical systems, statistical mechanics and automata theory. Some of the threads running through much of this work are the ability of neural nets to learn and self-organise,to generalise from training data, and to process information in other ways normally thought of as intelligent. This special issue is devoted to a class of networks which use, as their artificial neurons, units based on general Boolean functions and their probabilistic extensions. They are to be contrasted with nets whose nodes use weighted linear sums of inputs to define their activation. The terminology, 'digital networks' derives from the ability to implement the node functionality in hardware using Random Access Memories or RAMs which are, of course, a mainstay component in the construction of conventional digital computers. Digital networks are not a new field: indeed work on digital neural nets was well established before the recent resurgence of interest in semilinear neural networks, as marked by the publication of the influential volumes on "Parallel Distributed Processing" by Rumelhart, et. al. in 1986. We now give a brief overview that provides a perspective on the papers in this issue, meanwhile attempting to relate the work described to the three directions of approach outlined above. It is not, however, the intention here to give an exhaustive historical review and bibliography of the field.

Much of the early work on digital neural nets was carried out by Aleksander and collaborators (Aleksander, 1973; Aleksander and Stonham, 1979). The early phase engineering work on digital networks culminated in the commercially available WISARD pattern recognition device (Aleksander, Thomas and Bowden, 1984). This may be viewed as a hardware implementation of the n-tuple pattern recognition technique (Bledsoe and Browning, 1959; Bledsoe, 1961; Bledsoe and Blisson, 1962). Aleksander's insight was to realise that the underlying abstract machine could be implemented in semiconductor memory components that were just then becoming a possibility. Indeed, such devices were fabricated (Albrow, Aleksander and Noble, 1967) specifically for a prototype machine before the first commercially available RAMs. The WISARD machine consists of a set of n-input Boolean functions whose outputs are summed to provide an indication of the probability of a test pattern belonging to the discriminator's class. The key observation is that a Boolean function may be implemented as a look-up table in RAM. Training then takes the form of writing to RAM locations that are addressed during this phase. Recurrent nets of RAM devices have also been studied (Aleksander and Mamdani, 1968; Fairhurst and Aleksander, 1971; Masih, 1988; Martland, 1988; Gurney 1989) although this has usually taken the form of software simulation. A development of the Boolean function is the Probabilistic Logic Node or PLN (Myers and Aleksander, 1988) where more than two values are stored at each RAM location. These have primarily been used in multilayer feedforward configurations with Reward-Penalty style training algorithms.

The main thrust of the work described above was the training of nets to solve specific pattern recognition/recall problems. Recurrent nets of Boolean functions have also been studied from a point of view more akin to statistical physics, independent of any prior learning, as large assemblies of simple components that lead to complex dynamics. (Kauffman, 1969; Walker and Ashby, 1966; Martland 1987; Grondin et. al. 1983). In terms of the net's state-space, the behaviour is characterised by a transient part, where a state is visited only once, and a cyclic part in which each state in the cycle is repeatedly visited. Cycles may have many states or only a single, stable-state. This allows of an interpretation in terms associative recall from within a basin ofattraction to either a single 'memory' or a sequence of such memorised states.

The analogy between RAM nodes, developed in the engineering and computer science tradition, and biological neurons is, at best, striking and suggestive. Likewise, the superb perceptual, learning and cognitive abilities of real brains have served as an 'existence proof' to inspire engineers and computer scientists with the feasibility of intelligent, brain-style computing. However, if we are to be scientific, we should not be content with analogies: more exact comparisons with biology are needed. Progress in this direction has been made in the work of Taylor (1972) with his model of synaptic noise. This model was developed further (Gorse and Taylor, 1990) to include modelling of postsynaptic summation, cell surface geometry and axo-axonal interaction. Moreover, a close formal relationship was shown between Taylor's model of synaptic noise and a hardware-realisable probabilistic RAM or pRAM (Clarkson, Gorse and Taylor, 1989). Thus developments in the mathematics of probabilistic nodes have underpinned a convergence of engineering and biology. The dynamics of recurrent pRAM nets have now been studied using a Markov process analysis and small pRAM networks have been implemented in hardware (Clarkson, Gorse and Taylor, 1990). Both pRAMs and PLNs are probabilistic Boolean nodes: PLNs having discrete site values and pRAMs (theoretically) continuous site values.

An important question for engineers and computer scientists concerns the applicability to digital neural nets of the powerful learning algorithms which have been developed for net architectures consisting of nodes with continuously variable activations and link weights. By generalising the PLN to two types of device with continuous site values, Gurney (1989) has shown that these then avail themselves of an analytic treatment in terms of learning convergence. In particular, the Reward-Penalty scheme of Barto (1987) and Backpropagation (Rumelhart, Hinton and Williams, 1986) carry over, and techniques from control theory system identification may be used as a unifying framework for training algorithms. The importance of these proofs is that work on digital neural nets no longer need be considered as an isolated field; rather, the common applicability of the well-known training algorithms to digital and semilinear units allows the power of the digital approach to be appraised for its own merits.

The ease of implementation in readily available hardware, greater functionality, and computationally less intensive training (writing to RAM locations) provide adequate motivation to study digital nets as an alternative to the more usual nets of weighted semilinear units which have, traditionally, been at the heart of mainstream neural net research. The papers presented here develop the ideas outlined above and give theoretical analysis from several points of view. New training algorithms, analysis of net behaviour and capability, the connection with semilinear units and biology are all explored.

Several of the papers here deal, from an engineering standpoint, with new architectures, node types and training algorithms. Aleksander introduces a neural subsystem or cell assembly, the 'General Neural Unit' (GNU), which is then used as a building block for complex networks. Al-Alawi and Stonham describe a backpropagation style training algorithm for a certain class of feedforward nets of PLNs and a way of adapting network connectively to specific problems. Filho, Bisset and Fairhurst modify the functionality of the PLN with their introduction of the 'Goal Seeking Neuron' (GSN), and introduce a training strategy for nets of GSNs based on a search for unused node locations. Kerin and Stonham describe a self-organising network which uses nodes based on PLNs and with a clustering architecture. These last two papers develop and apply some of the ideas found in Grossberg's Adaptive Resonance Theory (ART) (Grossberg 1987). Austin describes his 'Advanced Distributed Associative Memory' (ADAM) and shows how it may be used to implement simple reasoning has embarked on an active new phase, with a strong emphasis on the properties of probabilistic nodes, supporting more powerful training algorithms and architectures than the earlier generation of deterministic RAM nodes. It will be important for the future of the field to consider how this emerging understanding of digital nets might be used to develop powerful technologies for new applications, as well as lraditional AI applications such as speech recognition, image interpretation, automated reasoning, and natural language understanding.

References

  1. Albrow, R.C., Aleksander, I., and Noble, P.J.W. 1967 "An Adaptable Universal m.o.s.t. Monolith," Electronic Communicator, Aug. 1967.
  2. Aleksander, I. 1973 "Random Logic Nets: Stability and Adaptation," International Journal Man-Machine Studies, 5, 115-131.
  3. Aleksander, I. and Mamdani, H. 1968 "Microcircuit Learning Nets: Improved Recognition by Means of Pattern Feedback," Electronics Letters, 4, 425-426.
  4. Aleksander, I. and Stonham, T.J. 1979 "Guide to pattern recognition using random-access memories," Computers and Digital Techniques, 2, 29-40.
  5. Aleksander, I., Thomas, W.V., and Bowden, P.A. 1984 "WISARD a radical step forward in image recognition," Sensor Review, July 1984, 120-124.
  6. Barto, A.G. and Jordan, M.I. 1987 "Gradient Following Without Backpropagation in Layered Networks," Proceedings of 1st IEEE International Conference on Neural Networks, Vol. 2, San Diego, Calif.
  7. Bledsoe, W.W. and Browning, I. 1959 "Pattern Recognition and Reading by Machines," Proceedings of the Eastern Joint Computer Conference, pp. 225-232.
  8. Bledsoe, W.W. and Blisson, C.L. 1962 "Improved Memory Matrices for the n-tuple Pattern Recognition Method", IRE Transactions on Electronic Computers, EC11, pp. 414-415.
  9. Carpenter, G.A. and Grossberg, S. 1987 "A Massively Parallel Architecture for a Self-Organising Neural Pattern Recognition Machine," Computer Vision, Graphics, and Image Processing, 37, 54-115.
  10. Clarkson, T.G., Gorse, D., & Taylor, J.G. 1989 "Hardware Realisable Models of Neural Processing," in Proceeclings of the 1st IEE International Conference on Artificial Neural Networks, 1989, pp. 242-246.
  11. Crick, F. 1989 "The recent excitement about neural networks," Nalure, 337, 129-132.
  12. Fairhurst, M.C. and Aleksander, I. 1971 "Natural Pattern Clustering in Digital Learning Nets", Electronics Letters,7, pp.724-726.
  13. Gorse, D. and Taylor, J.G. 1990 "A General Model of Stochastic Neural Processing," Biol. Cybern., 63, 299-306.
  14. Grondin, R.O., Porod, W., Loeffler, C.M., and Ferry, D.K. 1983 "Synchronous and Asynchronous Systems of Threshold Elements," Biol. Cybern. 49, 1-7.
  15. Gurney, K.N. 1989 "Learning in Nets of Structural Hype~rcubes," PhD Thesis; available as Technical Memo CN/lV144 Dept of Electrical Engineering, Brunel University, UB8 3PH, U.K.
  16. Kaufman, S.A. 1969 "Metabolic Stability and Epigenesis in Randomly Constructed Genetic Nets," Journal of Theoretical Biology, 22, 437-467.
  17. Martland, D. 1987 "Behaviour of Autonomous, (Synchronous) Boolean Networks," Proceedings of IEEE 1 st International Conference on Neural Networks, Vol. 2. San Diego, Calif.
  18. Martland, D. 1988 "Configurable Boolean Networks," Proceedings of nEuro'88.
  19. Masih, L. 1988 "Convergence in a Learning Network with Pattern Feedback," 4th International Conference on Pattern Recognition, Cambridge. plOO.
  20. Myers, C.E. and Aleksander, I. 1988 "Learning Algorithms for Probabilistic Neural Nets," 1st INNS Annual Meeting, Boston.
  21. Rumelhart, D.E., McClelland, J.L., and the PDP Research Group (Eds.) 1986 "Parallel Distributed Processing," MIT Press.
  22. Rumelhart, D.E., Hinton, G.E. and Williams, R.J. 1986 "Learning Internal Representations by Error Propagation," in D.E. Rumelhart, J.L. McClelland, and the PDP
  23. Research Group (Eds.) "Parallel Distributed Processing," MIT Press. Taylor, J.G. 1972 "Spontaneous Behaviour in Neural Networks," Journal of Theoretical Biology, 36, 513-528.
  24. Walker, C.C., and Ashby, W.R. 1966 "On temporal Characteristics of Behaviour in Certain Complex Systems," Kybernetik, 3, 100108.