#+TITLE: EuroClojure 2014, Krakow * Fergal Byrne, Clortex: Machine Intelligence based on Jeff Hawkins' HTM Theory - @fergbyrne - HTM = Hierarchical Temporal Memory - [[http://fergalbyrne.github.io/euroclojure/euroclojure.html][Slides]] ** big data - big data is like teenage sex - noone knows how to do it - everyone thinks everyone else is doing it - so everyone claims to be doing - (Dan Ariely) ** machine learning is important - people don't trust other people - they have their own agendas - so they place too much trust in machines ** asimov's take - we gain knowledge faster than we gain wisdom - applies to human knowledge - applies to data: gathering data is easy, drawing conclusions is not ** a problem in neuroscience - rate of papers published is growing exponentially - 2013: 1 every 32 minutes - 2014 so far: 1 every 17 minutes ** can AI learn from neuroscience? ** Jeff Hawkins' goals in HTM - Study the neocortex and establish its principles - open sourced NuPIC in 2013 ** neocortex - the wrinkly part at the surface of the brain - grey matter: processing - white matter: wiring - about 2mm thick, 10cm^2 in area - 30-50MM neurons - 1G connections - *hierarchical* - *uniform* - ie all looks physically the same - all regions have the same algorithm ** 6 key principles *** on-line learning from streaming data - up to 10 million senses feed the brain - we don't (can't) store this data - we build models from live data - models constantly updated *** hierarchy jof regions - sensory data enters at the bottom - models are built in every region - things change more slowly as you go up - hierarchy enables sequences of sequences - seq of waves - seq of phonemes - seq of words - seq of sentences - hierarchy works upwards and downwards *** sequence memory - all sensory data involves time - sequence memory allows predictions - structure in data elaborated over time - sequences can be c *** sparse distributed representations - in each region, many neruons, few active - SDRs represent spatial patterns - fault-tolerant, semantic ops, high-capacity - key to understanding & building intelligent systems *** all regions are both sensory and motor - behaviour provides context for sensory data - structure in model navigated via behaviour *** attention - use attention to manage the neocortex - planning and previsualisation - whole subhierarchies can be switched on and off ** layers of neocortex - from molecular upwards - around 5 or 6 *** neurons - distral dendrites detect coincidence of incoming activity from neighbouring cells - you don't just see what you're seeing now, you predict what you're going to see next - (reality is much more complicated, but this algorithm is sufficient to explain a lot) ** clortex *** background: numenta's nupic - in dev since 2005 - partially implements HTM/CLA - python/c++ - open source **** strengths - skilled dev team - eat their own dog food (grok uses nupic) - operates on subset of HTM/CLA principles - tunable using swarming on your data - works well on streaming scalar data (eg machine-generated) - great community -- http://numenta.org **** limitations - codebase has evolved as theory has developed - difficult/scary to rewrite for flexibility - OO with large, coupled, classes (~1500 LoC per class) - need to swarm to find parameters, no real-time control - not easy to extend beyonnd streaming scalar use case *** clortex requirements - directly analogous to HTM/CLA theory - transparenntly understandable source code - a neuroscientist should be able to read & review code - directly observable data - sufficiently performant - useful metrics - appropriate platform - portability - scalability *** architectural simplicity - first role: be useful! - best software is that which is not needed at all - human comprehension is king - if people can't understand your code, your code is not finished - unit tests are not sufficient in themselves - machine sympathy is queen - software is a process of R&D - software development is challenging & intellectual - more science than engineering - engineering: you have a good model already, you just have to plug in the particular parameters - science: there are a bunch of unknowns which you have to learn & understand *** #1: Just use data! - maps, vectors, sets - all done in a one-page datomic schema *** #2: Clojure & its ecosystem - clojure data not domain objects *** #3: russ miles' life preserver - everything either "core" or "integration" - core: a datomic database for the neocortex - core: each "patch" of neurons is a graph - integration: algorithms, encoders, classifiers, SDRs *** key clj libs & tools - datomic (+adi) - quil/processing - incanter - lein-midje-doc for literate documentation - hoplon-reveal-js for presentations - lighttable ** review - Big Data isn't just Machine Intelligence problem - HTM is exciting ** links - http://numenta.org - http://inbits.com - https://github.com/fergalbyrne/clortex - writing a leanpub book * Logan Campbell, Clojure at a Post OFfice ** history: - was at clojure user group - a guy turns up and says he's hiring a team of clojure developers - he was at Australia Post - a million lines of Java worked on by a team in India - wanted to bring it back in-house ** project: digital mailbox - big companies spend a lot of money sending out bills & junk mail - product to seamlessly replace that workflow - switch from physical mail to cheaper model - consumer can sign up to receive water bill online - I was brought on as the "clojure expert" - (I'd been playing with it for a couple of years) - drama: - the people they could hire: - really experienced java devs - keen on FP - they said as they were hiring "you might be doing clojure or you might be doing scala" - first few people were scala fans - scala v clojure battles - "we need static typing" - "we need OO for domain modelling" - "clojure is slow" (?) - "what framework do you use?" - "we need static typing? okay, we'll use core.typed" - domain modelling: - when people are used to domain modelling in OO, telling them to just use maps feels like a cop-out - records + protocols kind of feel like classes - wasn't til I showed them code I'd written and comparing it with their code that they realized that you can just use maps - online scala course - we did it as a team - I also did the exercises in clojure - did one exercise three different ways in clojure - conditional - match - stream processing - showed them my solutions - they already understood the problems because they'd solved them themselves - clojure performance was a surprise, because I'd come from ruby (!) - clojure is /fast/ - there was an underlying feeling that "we need scala for performance" - I'm a consultant, so was happy for the team to make the language decisions - "if you're keen on scala, let's find out a way to pitch it to management" - web stack: kept hearing "async async async" - felt like premature optimization - but still we used http-kit - benchmark started to allay fears that clojure was slow ** feature: make a payment on a bill - not necessarily a full payment : POST /bills/:bill-id/payments : Session: user-id : Post Data: amount - GET credit card token for user - POST request to payment gateway - GET how much left to be paid - if payment succeeds: display amount remaining - if payment fails: display error ** candidates solutions - synchronous promises - promise monad - lamina - etc etc ** solution 0: synchronous - http-kit's requests return a promise - just @deref the promise (blocks the thread) ** solution 1.1: promise monad - ~do~ is aware of promises - doesn't block thread, but waits for promise to be executed before continuing - felt natural way to write with promises - but incorrect: too much waiting, no concurrency ** solution 1.2: promise monad let/do - ~let~ to define promises - ~do~ to pseudo-block on them - introduces correctness but reduces readability ** solution 1.3: let/do/do - okay, let's step away from monads ** solution 2: (?) ** solution 3: raw promises - ~when~ to explicitly wait for a particular promise ** solution 4: raw callbacks - not viable - would have just written a hacky little promise library ** solution 5: core.async: - great! same shape as synchronous code, but correct concurrency ** solution 6: lamina - didn't feel totally suited to the situation ** solution 7: meltdown (LMAX disruptor based) - not appropriate ** solution 8: pulsar promises - looks exactly the same as the synchronous code, except for one character - pulsar rearranges your code at the bytecode level - uses JVM agents (normally used for tracing/debugging) - pass a fn to one of pulsar's functions - turns synchronous code to async code ** solution 9: pulsar actors - not appropriate ** winners - 0: synchronous - 5: core.async - 8: pulsar ** scala solution, for comparison - scala futures (basically promises) - all monadic - I don't understand it entirely - concise - battle of the benchmarks, fastest first - pulsar-async - pulsar-sync - core-async - raw-callback - scala-play-future (significantly less than others) ** CQRS (command-query responsibility segregation) - want fast reads - reduce number of queries - don't want to have to update write code every time we add a new reader ** structure - service A → cassandra → service B - custom triggers in cassandra in clojure (just drop in the .jar!) - publish to rabbitmq - notify index maintainer - write index to cassandra - service B reads from cassandra ** cassandra triggers - can just throw the clojure jar in there - everything is byte buffers - you need to know the type of all the fields out-of-band - not self-describing data at all ** microservices - I thought we would have a user service and a provider service and a mail service - but this gets tricky when you want data about users and providers - you need to split things much more fine grained - user service → - authentication - multi-factor auth - authorization - user profile - password reset - does it belong in user profile? - there's a bit of workflow here - send out email - get user to click link - enough to warrant its own service - drama: needed to talk to systems team to deploy - I did things badly - I didn't get anything into production in my 6 months there - systems team: we need monitoring and config and stuff - if we'd had something early on which had gone through these barriers, we would have had much less stress - benchmarks end petty arguments ** Q&A *** can you share some experience with monitoring & resilience? - appdynamics - classnames are expected to be java-style class names - clojure ones are close enough - clj-metrics to expose more high-level metrics - requests/second from ring - number of bills paid - appdynamics could pick it up from jmx - nomad for configuration *** with http-kit+core.async, what happens when server dies and there's loads of threads? - bottleneck was amount of memory - when server runs out, it slows down a lot - way to get around that is to monitor resources on your machine and ideally have autoscaling *** were the scala guys finally writing clojure in the end? - we have one person still hardcore for scala, but sees the merits of clojure - a few who did the online scala courses are clojure folks now - people who come from the java world of static typing feel they need that - but now they've written code that actually works, they're more comfortable with that now * Tom Hall, Escaping DSL Hell by having parens all the way down - @thattommyhall ** DSLs - languages made for specific purposes - config mgmt - science - learning - distinction between: - internal DSLs: embedded in another language - external DSLs: implemented in another language ** problems with puppet - zen of python: - namespaces are a honking great idea, let's do more of them! *** puppet namespaces - Exec['install'] in two different modules will result in a naming collision - fail :( - end up with Exec['tom::install'] but this is a hack *** iteration - file type lets you pass in an array - nagios_host doesn't - iteration is responsibility of type, not language - as far as I know *** but you need to know ruby anyway - if you want to extend puppet, you need ruby - if you need to know ruby, why do we bother with the puppet DSL in the first place? *** experimental features: lambdas and iteration - any language where lambdas arrive late is not a good language ** ansible - just YAML - oh wait, I might want to iterate - oh wait, I've got embedded ginger templates in my YAML strings - what's the scope of names in my templates? ** if you give people a "language" they will expect loops - maybe lambdas - probably namespaces - this has been done before ** chef gets it right - it's embedded in ruby - you get iteration and namespaces from ruby ** teaching people to program - if you design a language: - you need a parser, which is hard - you need an interpreter/compiler, which is hard - if you embed it, you get that stuff for free ** geomlab - minimal language for teaching - talks about pictures - intro to FP - gets you into recursion early on - ~man $ woman~ - "next to" - ~man & man~ - "on top of" - ~(man $ woman) $ tree~ = ~man $ (woman $ tree)~ - ~man $ (woman & tree)~ -- scales nicely to get a nice aspect ratio - learn about operator precedence - de morgan's laws - although not always held, due to scale - define functions : define manrow(n) = manrow(n-1) $ man when n>1 : ~ manrow(1) = man - builds up to an escher tiling - but once you've done that, where do we go? - only exists in this sim - if you want to extend it, you need java - "I'm really excited about FP now, but I've got nowhere to go" ** what if we did it in clojurescript? - let's use 'below and 'beside instead of $ and & - ~(below man woman)~ - ~(beside tree star)~ - http://cljsfiddle.net/fiddle/thattommyhall.geomlab.demo - let's say I want to change ~man~ -- what does it mean? - it's implemented in the same sort of language - I can see there's a url in there where I fetch an image from the internet - I know recursion, because I learned that from the geomlab exercises - I can extend the language itself ** science languages - R - wolfram alpha - maple - matlab - these things just aren't very good languages, even if they are good at their domain ** another problem with DSLs - netlogo - http://ccl.northwestern.edu/tortoise/2013-10-25/Ants.html - If you're based on applets, and Oracle drops applet support, you find you need to port your whole language to a new platform (in this case javascript) - again, reimplement in clojurescript? - anyone interested in hacking on this with me? ** conclusion - you probably don't need to make a new language - if you do it will probably be rubbish - at least for a while - think about power and reach - *you should embed /deeply/ into clojure* ** links - http://twitter.com/otfrom - http://cljsfiddle.net/fiddle/thattommyhall.geomlab.core - http://cljsfiddle.net/fiddle/thattommyhall.geomlab.demo - http://cljsfiddle.net/fiddle/thattommyhall.geomlab.bruce - http://www.complexityexplorer.org/ - http://cljsfiddle.net/fiddle/thattommyhall.ants.core - http://ccl.northwestern.edu/tortoise/2013-10-25/Ants.html ** Q&A *** what makes a good first language? - clojure needs a better day 0 story - at some coder dojos where I've taught kids, some don't even know about files and folders - so if you say "open a terminal, cd into a directory" you've lost them - and it's not their fault *** have you had any kids look at your examples here? - I've done the geomlab example - otherwise this is all a recent exploration - errors in cljsfiddle are not reported well - again problematic for day zero * Mathieu Gauthron, JVM-breakglass - using a clojure REPL to troubleshoot live java/JVM processes - http://slides-euroclojure2014.matlux.net - when you see fire, you break glass - when your jvm process is on fire, you use JVM-breakglass ** troubleshooting a java application - debugger - only powerful when you can narrow down the problem to a series of breakpoints - when the problem is a race condition, it will change the nature of the problem you're studying - log/print statements - you need to plan before compilation - when the problem is in production, it might be too late - jmx - again, you need to plan for it in advance - ad-hoc interactive mechanism ** what is jvm-breakglass - open source - integrates with any jvm process - console onto a jvm process ** main features - interactive prompt - see inside private members - call arbitrary methods - create new object instances - create new classes - monitor object state - no need to use clojure to develop the app ** how does it work? - jvm-breakglass runs inside the JVM and starts an nrepl server - you can then connect using an nrepl client (eg lein) ** how to use it? - add it to your maven dependencies - add an entry point (as a ~~ or in java code) - connect with ~lein repl :connect localhost:1112~ ** demo (enterprise application) - tomcat JVM - employee/dept data structure - report generation - java/spring mvc webapp - jvm-breakglass - spring data - in XML, naturally *** homepage - oh no! one of the reports isn't working? - "list employees in london" is empty - but we know that employee Mick Jagger lives in london - what's going on? *** breakglass to the rescue - view environment: - current directory, System/getProperties - view conf directory - list all loaded Spring beans - instrospect into object private members - ~bean~ builtin fn - ~to-tree~ to do so recursively - view methods or fields for a given object - redefine a class - in this case, ~(proxy [Address] ["1 Mayfair", "SW1", "London"] (getCity [] "London"))~ to define the new version, overriding a method - ~(.setAddress (:Mick employees) address)~ to inject it into the live data ** take a step back - remember what it's like to be a java programmer? - working with jmx beans and suchlike to try to understand why production is down - this stuff looks like magic ** Q: how do you convince production people to put nrepl server in place? - short answer: impossible - that's not how you present it - either you do it sneakily (that's bad), and only pull the trump card when the team is desparate - or you convince the team that it would be useful in the UAT environment, and "of course it's never going to be used in production" - ** Q: have you considered a high-level switch that would prevent you mutating anything in the host application? - don't know how you'd be able to do that - have been thinking about it - maybe using clojail - kind of defeats the point ** Q: have you tested this with a scala app? - haven't tried - I've reverse-engineered the java bytecode, and it's readable - as long as you know how it compiles, it seems reasonable ** Q: you were using methods like get-obj and passing string name. how does breakglass know which object to get? - eg if you have multiple instances of Department, how does it know *which* department? - in Spring it's a Spring bean which is named - if you're not using Spring, what's your entry point? - when you create your NreplServer to enable jvm-breakglass, you can add your entry points there - ~new NreplServer(port).put("department"),myObject);~ - static methods & fields can be used too * Gary Crawford, Using Clojure for Sentiment Analysis of the Twittersphere - slides: http://www.slideshare.net/garycrawford/using-clojure-for-sentiment-analysis-of-the-twittersphere-euroclojurhttp://www.slideshare.net/garycrawford/using-clojure-for-sentiment-analysis-of-the-twittersphere-euroclojur - leiningen versus the ants, carl stephenson - leiningen versus apache ant? - clojure versus java? - FP versus OO? ** stratified medicine - determine the best treatment for someone based on their genetic makeup to manage their chronic disease ** sentiment analysis - Paper: "Twitter mood predicts the stock market" - predicted Dow Jones average through monitoring tweets - people who suffer chronic disease tend to be neurocompromised - what would normally be a minor illness can prove fatal - can we use twitter to predict spread of disease? ** so we tried - score tweets for flu symptoms - the data science wasn't very difficult - but scaling it was - 30 million geo-tagged tweets sent from UK - couldn't scale, even with - HDFS/hadoop - mongo/aggregation - mongo/mapreduce - postgres ** how can we do fast, real-time analytics of social media? - application: how do people feel about Scotland's independence referendum? - data increases in value as we analyse it - tweets - analytically prepared data - analysis - insight - predictions - the raw data isn't what you care about - don't store the raw tweets, only store the analytically prepared data - stored in redis using ptaoussanis/carmine - it has great support for bitmaps ** example - ~(car/setbit sentiment tweet-id 1)~ - ~(car/bitcount "SCOTLAND")~ -- tells me how many tweets have mentioned Scotland - how many people in england are happy? #+begin_src clojure (wcar* (car/bitop "AND" "ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire "ENGLAND&JOVIALITY" 10) ;; don't keep the data longer than 10 seconds (car/bitcount "ENGLAND&JOVIALITY")) #+end_src - further: "how many people in Scotland are tired or grumpy?" ** getting the data in - adamwynne/twitter-api - you can specify you only want tweets from a certain geographical locality with a bounding box - but this is literally a rectangle - need it around Europe - LMAX-Exchange/disruptor to communicate - journaling - syncing - business logic *** what sentiment? - this is hard! - "I'm loving #EuroClojure! :D" - Positive Affect: enthusiastic, active, alert - Negative Affect: subjective distress - actually two separate dimensions, not opposites - Watson et al, 1988 - PANAS - then PANAS-x - then PANAS-t - accounts for bias on social media - outlines sanitisation - validate against 10 real events **** sanitisation - https://github.com/dakrone/clojure-opennlp - get rid of spam - account for text speak - account for emoticons and emoji - word stemming (or lemmatisation) - part of speech tagging *** where? reverse geocoding - don't want to rely on external services - don't want heavy IO - don't want round trips to database - accuracy not too much of a concern - we already lose accuracy in interpreting the sentiment of the tweet - convert a map of the uk to colours: - look up geocode coords in map - check colour → get country code - problem: the world is a sphere - projecting a sphere onto a rectangle - prior art in d3.js - use JavaFX to exploit it *** when? - there's a lot of seconds in a day - and even more seconds in a year - really not interested in seconds anyway - want to group tweets by minute - and also group by hour - and also group by day, and month, and year ** why? - why are we doing this? - online social media are surveillance - the line between public and private is becoming blurred - if we don't need data, we shouldn't collect it - in this example: - we're never more granular than country - we're never more granular than overall sentiment - we're never more granular than minute - hopefully this is enough to prevent anyone being identified - Datensparsamkeit ** Q: have you used Storm for this? - no ** Q: any preliminary results on the Scotland referendum analysis? - I've had more luck with tech than data science? ** Q: which way should we vote? - haha ** Q: how do you verify your results? - it's very crude at the moment? * Paul Ingles, Multi-armed Bandit Optimisation in Clojure - @pingles ** problem statement - product optimisation cycles are long, complex, and inefficient - the multi-armed bandit model shows lots of things we're getting wrong - eg: online newspapers - fundamentally human-led, editorially-led - people behave irrationally - Dan Ariely & Daniel Kahnemann - (@philandstuff suggestion: Stuart Sutherland, Irrationality) - economist subscription options 1. online $59 2. print $125 3. print & online $125 4. the ridiculousness of option 2. makes option 3. seem more reasonable - need machines to optimise at scale; but need humans to provide stuff only they can - running RCTs to optimise sites - doing so on a continuing basis - measuring big effects work with small numbers of participants - but measuring small effects requires ever larger numbers - to the extent that you can only run ~12 experiments a year - which is not really good enough ** Bandit strategies can help - a product for procrastinators by a procrastinator - Product: Notflix! - video website - http://notflix.herokuapp.com/ - shows 3 different videos - show good videos at top of page, and less good at bottom - show best possible thumbnail for each video - optimising with multi-armed bandits - optimising order and thumbnails ** multi-armed bandit problem - slot machine = one-armed bandit - problem: you have a bunch of money you want to "invest" in a casino - you have a number of different machines to play - each machine has a different probability of reward - you don't know what that probability is up front - need to balance "exploration" and "exploitation" - ie learning about the world vs using that knowledge to maximise income - analogy: trying new foods out vs sticking to what you like ** bandit model - number of *arms* {1, 2, ..., /K/ } - number of *trials*: 1, 2, ..., /T/ - *rewards*: {0,1} - /K/-headlines - options of different text - /K/-buttons - options of button text, colour, etc - /K/-pages - whole page redesigns - explore this space with notflix ** bandit strategy #+begin_src clojure ;; choose which arm to pull (defn select-arm [arms] ...) ;; update arm with feedback (defn pulled [arm] ...) (defn reward [arm x] ...) (defrecord Arm [name pulls value]) #+end_src *** ε-greedy - "hello world" algorithm - generally exploit - ε (epsilon) is the rate of exploration - eg if ε = 0.1, your strategy is: - with probability 10%, try a random arm with equal probability - with probability 90%, try the best arm based on current knowledge - if ε = 0, always exploit; if ε = 1, always explore - example with bernoulli-bandit #+begin_src clojure (bernoulli-bandit {:arm1 0.1 :arm2 0.1 :arm3 0.1 :arm4 0.1 :arm5 0.9}) #+end_src - with ε=0.2, you converge faster on the best arm - but ε=0.1, you exploit it more when you find it - once you've found the best arm, you should be able to double down - ie explore more at the beginning (when you have least knowledge) and less at the end - lots of extensions to ε-greedy to factor things like this in *** Thompson sampling - Arm model - Θ_k: Arm k's hidden true probability of reward (in range [0,1]) - can build a distribution for Θ_k based on current knowledge - small number of pulls means wide distribution; large number means narrow distribution - captures uncertainty in value of Θ_k - each iteration, take a random sample from each distribution, take the largest sample - algorithm naturally balances exploration/exploitation trade-off - the more it learns, the narrower the distributions get, and so the more likely it is to choose an arm with a higher expected value - incanter example - Thompson-sampling example with same Bernoulli-bandit from above - compared with ε-greedy, explores much more much earlier, and exploits much more later on - considered optimal convergence - we can use it to rank things (not just select) - take a sample from each arm distribution, then order arms by that value - in notflix, can use for ordering the videos we show ** applied to notflix - video rank bandit - for each video, a thumbnail bandit - at the end, the best video should be at the top - and each video should show the best thumbnail ** results - videos, worst to best - "hero of the coconut pain" - "100 Danes eat 1000 chillies" - "3 year-old with a portal gun" - thumbnail bandit data - "we built a fictional but /amazing/ product" ** links - [bandit/bandit-core "0.2.1-SNAPSHOT"] - https://github.com/pingles/bandit ** Q: this model assume bandits have same probability through time - can it readapt? - Thompson sampling does adapt - it won't change back as quickly ** Q: isn't there an interaction between the two bandits? - if the thumbnail is crappy, they might not click the video - made an assumption about this - in general, if you leave it running over time and let the evidence build, it should be fine in the long run - but that is definitely a flaw * Tommi Reiman, Schema and Swagger to improve your web APIs ** super simple web api in clojure - just using compojure - "sausage" as example data - ~PUT /foo/sausage/:id~ - example: - in Java: immutable value object - in Scala: case class - in Clojure: - free-form map? - constructor fn with bunch of validation? - prismatic/schema! ** prismatic schema - define structure of sausage - then call ~s/validate~ to validate - schema can define functions #+begin_src clojure (s/defn get-sausage :- (s/maybe Sausage) [id :- Long] (@sausages id)) (s/defn ^:always-validate get-sausage2 :- Sausage [id :- Long] (@sausages id)) #+end_src *** schema coercion #+begin_src clojure (defmodel Pizza {:id Long :name String :price Double :hot Boolean (s/optional-key :description) String :toppings #{(s/enum :cheese :olives :ham :pepperoni :habanero)}}) #+end_src - allows slurping JSON data, but imposing extra types - eg above we can slurp toppings from a JSON array into a Clojure set rather than a vector *** double schema - loose schema for first input - ~(def Customer {...})~ - tighter schema for validated input - ~(def ValidCustomer (merge Customer {...}))~ *** schema selectors - accept but remove unrecognised params with ~select-schema~ *** generative schema - generate random orders for test data - [[https://gist.github.com/davegolland/3bc4277fe109e7b11770][davegolland/generative-schema.clj]] -- demonstrates how to convert schemas to test.check generators *** contribs - sfx/schema-contrib - cddr/integrity ** swagger - a specification for describing, producing, consuming, visualising RESTful web services - https://helloreverb.com/developers/swagger - existing adapters - clojure options: - octohipster - swag - ring-swagger - compojure-api - fnhouse-swagger - endpoint definitions in JSON - data models as a JSON Schema - swagger UI - visualises the API - code gen - no clojure support yet (anyone?) - swagger-socket - run it all on top of websockets ** ring-swagger - https://github.com/metosin/ring-swagger - JSON-Schema has some dates - but prismatic/schema will never support dates, as it's more generic - higher level abstractions on top of swagger, but nothing for the web developer ** compojure-api - an extendable web api lib on top of compojure - macros & middleware with good defaults - schema-based models & coercion - ~GET*~ macro to define input and output schemas ** fnhouse-swagger - prismatic/fnhouse - launched at clojure/west - ~defnk~ with metadata → annotated handler - fnhouse-swagger - metosin/fnhouse-swagger ** summary - schema is an awesome tool - describe, validate, coerce your data - building on top of ring-swagger - compojure-api → declarative web apis - fn-swagger → meta-data done right - or do your own! - kekkonen.io - CQRS-lib * Renzo Borgatti, The Compiler, the Runtime and other interesting beasts from the clojure codebase - http://twitter.com/reborg ** an amazing growth: - mar 2006: first commit - oct 2006: 30k loc (7 month old) - oct 2007: clojure announced! - oct 2008: invited to Lisp50 to celebrate 50 years of lisp - May 2009: 1.0 + book! - now: almost 90k loc ** initial milestones - apr 06: lisp2java sources - may 06: boot.clj appears - may 06: STM first cut - june 06: first persistent data structure - sep 06: java2java sources - aug 07: java2bytecode started - right after: almost all the rest: refs, lockingtx ** drew on lots of sources of knowledge - collection of papers ** high-level view: - ~(def lister (fn [& args] args))~ - read → analyse → emit/compile → compile - although the lines between the stages get blurred at times ** reader - takes stream, returns data structures - PersistentList, Symbol, etc ** analyser - input: data structure - output: exprs - DefExpr - Var - FnExpr - Sym - PersistentList - FnMethod - LocalBinding(Sym("args")), - BodyExpr - PersistentVector - LocalBindingExpr ** Emission - bytecode generation for Exprs - prerequisite for evaluation - emit() method in Expr interface - Notable exception: called over ?? ** Evaluation - transform Exprs into their "usable form" - eg - new object - a var - namespace - FnExpr is just getCompiledClass().newInstance ** Compilation - Usually coordination for emit - Compiler.compile namespace -> file - ... ** Emit - input: Exprs - output: bytecode ** monsters! *** RT - this is how the RT class gets initialised: the first time it gets referenced: #+begin_src java final static private Var REQUIRE = RT.var("clojure.core", "require"); #+end_src - simply referring to it here causes the static initializers to run - RT has a *lot* of behaviour in static initializers - inside it is the ~doInit();~ call - which loads all of ~clojure.core~ - all just from referring to RT in some otherwise unrelated class! *** Compiler - inner classes for each Expr type *** LispReader - inner classes for each token you might encounter - ~~ - sets up reader macros - ~macros~ and ~dispatchMacros~ (latter for ~#{~ ~#(~ ~#_~ ~#^~ etc) *** analyze() - not a class, but a family of methods - ~analyzeSeq~ - ~new ConstantExpr~ - ~MapExpr.parse~ - FnExpr.parse - invokes the compiling phase during parsing phase *** emission - ASM lib used to generate bytecode - FnExpr.emitMethods() - generate a method for each of the arities of the function *** other beasts - LockingTransaction and Ref ** DynamicClassLoader - ~clojure.lang.DynamicClassLoader.findClass(String)~ - ~RT.classForName()~ - ~Compiler$HostExpr.maybeClass()~ - Class.forName() goes up the hierarchy of classloaders and asks each what they know - an instance of DynamicClassloader is created for each namespace - and also for each form - (this is true for the bootstrap phase; not always true eg in AOT (ahead-of-time) compilation) - supporting dynamicity - in defineClass: - ~classCache.put(name, new SoftReference(c,rq));~ - in findClass: - ~Reference cr = classCache.get(name);~ - SoftReferences are used to save PermGen, since if we redef a var we don't want it to keep consuming PermGen ** Bonus: clojure was initially implemented in lisp - ~1600 loc to implement read, analyse, compile, eval - although emitting Java code, not bytecode - was also generating C♯ ** Q: some things in bytecode can't be expressed in java - is there anything which clojure generates which can't be decompiled back to Java? - I'm pretty sure yes, but not sure exactly what - Rich: - locals-clearing - constructs which use goto (which exists in bytecode but not Java) * Rich Hickey, the insides of core.async channels ** aside: here's what clojure looks like in a good IDE - (ie IntelliJ) - yes, Compiler.java is massive - but if your IDE has a structure editor, you can navigate them all easily - it's all in one file because I don't want 300 files ** aside2: the classloader has a cache in a branch - fast-load branch ** warning! implementation details ahead - subject to change! - informational only ** the problems - single channel implementation - for use from both dedicated threads and go threads - simultaneously, on same channel - alt and atomicity - Java CSP libraries often didn't support alt well - it's tricky to do atomically - multi-reader/multi-writer - concurrency - construct deals with the ick of threads and mutexes - (this talk: focus on JVM impl; JS version has less of these issues) ** API - ~>!~ ~>!!~ ~put!~ ~alt!~ → channel → ~!~ ~>!!~ ~put!~ ~alt!~ → ~impl/put! [val handler]~ → channel → ~impl/take! [handler]~ → ~1s) - converges slowly, but: - approaches ~23 ms ** now what? - typescript, dart? - these are under the opinion we want to build the same broken type of stuff - cljs: we can build things radically simpler *** React - library from facebook - other libraries have a deep-seated notion that everything is mutable - angular, backbone, ... - react is different: it has a functional mindset - the virtual DOM evolves from one value to the next - clojurescript allows fast diffing between these values - react will do the right thing - react has completely taken over the cljs world - Om - reagent - quiescent (much thinner) - reacl *** Om - Om was an experiment to show that representing app state a single global value was a good idea - this had been done before in other areas: - databases - server-side - we're not going to make interfaces that people haven't seen before - prismatic's blog post about moving to Om - simple components which don't interact in crazy ways *** Goya - by Jack Schaedler (sp?) - ui dev for ableton - "we can do real undo" - Jack saw this and wondered if it would scale - Goya: pixel editor - surface: immutable vector - gets undo without adding complexity to app - get almost unlimited number, without loss of performance - github: jackschaedler/goya - his app is complicated! - the UI is complicated - but cljs eliminates /unnecessary/ complexity - how much memory does his app use? not much - (aside: use google chrome dev tools!) ** innovate! - model story needs work - js MVCs backbone/ember/angular - notion of a model on the client - you can do operations on it - nothing particularly compelling for this in the react space - DataScript - export some elements of the datomic api to the client - store your data in a flat way - sensible query api over it (queries on trees aren't so fun) - datomic allows you to ask for entities - lift a tree out of the flat database - react model can be further improved - addressability - immutable everything - they have to convert styles and DOM attributes back to javascript objects which have to be walked - (one benefit of react: it's facebook's problem 😺 ) ** Q: is it possible to implement Om all in clojurescript using a macro? - I suppose it could - you might want to compose things dynamically, and macros are static - you have to be concerned with the amount of code that a macro generates - I would not pursue that idea ** Q: is there a community place for shared Om components? - I'm not going to spend much time on it - if Om needs to be improved to make this happen, I will do that - you want to be able to use other people's code without jumping through too many hoops - things get tricky with events & communication between components - need some agreement on how people communicate between components ** Q: what's your vision for cljs 1.0? how can we help with the yak shaving? - basic things like sharing code - shared analysis over clojure and clojurescript - would open up a lot of tooling - eg linter to lint both languages - would like infrastructure for tooling to be much better - when you go to 1.0, people lock to that version and are slow to move off it ** Q: are you seeing much evidence of cmd-line or server-side cljs? - most people doing it are doing node.js ** Q: when is cljs going to be self-hosting? - it's not that we don't want it - we're keen on self-hostability/bootstrappability, if not self-hosting - nice to remove the JVM dependency - eg lighttable might not want it - it's last-mile stuff at the moment - which isn't that fun - and I don't personally need it so I won't work on it ** Q: do you forsee a pure cljs version of react? - if someone wants to shave that yak, that would be awesome - if the system is immutable all the way down, the optimizability explodes * Ali Asad Lotia, Why devops needs Clojure - @aalotia ** background - was a dev who had helped get stuff to prod - our ops person left - they asked me to fill in - I said "okay, as long as you hire a replacement soon" - they didn't arrive - I missed being able to write code ** problem - we're exec'ing a jar, and it keeps taking 3s - I saw an opportunity to write a very simple noir app - much improved performance - people were impressed, asked to see the source code - "what is this clojure thing, and why did you use it?" - I'm not a seasoned Java dev - moved to another company: ** Beamly - TV focused social network - smart TV planner - personalised TV Magazine - availability *** behind the curtain - AWS: us-west, us-east, eu-west - milli-services (ie not quite µservices) - scala - node.js *** my team - build/release automation - but we /don't/ do deploys; we just enable them - persistence - platform performance/metrics/logs - core libs ** deploys in the mutable days - generate build artefacts - define config in puppet - deploy artefacts - deploy config ** phoenix servers - base server images, with some configuration changes - relatively short-lived - didn't name them or worry when they were switched off ** disconnected dev and ops - zed shaw: - "maybe you use a language like lisp that pretends the computer is some purely functional fantasy land with padded walls for little babies" - actually, yes I do ** immutable servers - kill server for every deploy - package new server images (AMIs) in order to deploy new version ** Requirement: examine server images - aws console - some config but manual and not all info - python + boto: - just got back a list of objects - we know there's more available! we saw it in the console! - clojure + amazonica - it just gave us *data* back! - data trumps objects every time for this kind of use case - console → cli → sdk → repl/scripts ** repls are awesome - exploration of APIs - minimise context switching - instant feedback - data rich (or richer, at least, in some cases) ** team reactions - "Soooo many brackets!!!" - I don't see them anymore -- paredit deals with it - "How do I iterate over this?" - why do you need to iterate? what are you trying to do? - "I want to change this value" - again, what are you trying to do? - "Wow, this is really powerful" ** Offloading state - Immutable servers - http://martinfowler.com/bliki/ImmutableServer.html - pass the buck to a service someone else manages - application data - metrics - but when you do autoscaling, it takes some problems away but gives us other problems - provider defined data model - clojure was a great fit for managing autoscaling groups - all the information we needed was made visible by a single clj fn - in a repl, with ad-hoc tasks, having some clojure code you've written and evalling it is really powerful ** Observing platform performance - knock-on/trickle down effects - sensu handlers limited *** riemann - http://riemann.io - had used graphite - very data-poor - riemann gives you a clojure map, which is a much richer model - embedded REPL - overridable - extensible - responsive primary author ** tracking our services - zion - system knowledge base - who owns which service? what do I do when alert X fires? - component details ** infrastructure as data - config - metrics - logs - we had powerful ways to analyse this data without having to resort to glomming 500 scripts together - we have a single language which is superlative - I sit next to extremely good Scala devs and ask how they would do it - "I'd write a case class" ** future work - analyse logs + metrics - catch and correct misconfigurations - scripts with upcoming fastload? - if clojure fastload is fast enough that we don't have to worry about startup time, could it replace some of our python scripts? - cyanite to replace graphite - cassandra/clojure - "lisp isn't a language, it's a building material" - Alan Kay ** clojure summary - pros - core data structures - data manipulation - community - #clojure and #ldnclj on freenode - people accept PRs, give real feedback - projects move - shared aesthetic ** refs - martin fowler posts above - mcohen01/amazonica - pyr/cyanite ** Q: can you share more info about zion? - we will when it's in decent shape - too coupled to our particular environment right now ** Q: graphite data poor? can you elaborate, particularly with reference to storage backend? - how data is stored is poor - all you get is an arbitrarily long key name (hierarchical) - a timestamp - a single numerical value - with Riemann, you can add arbitrary tags to the events - persisting them -- don't have a great answer - looking at influxdb - store time-series data with a richer data model * Leonardo Borges, Taming Asynchronous Workflows with Functional Reactive Programming - who has used Reactive extensions? - do you think it's FRP? - currently writing "Clojure Reactive Programming: RAW" - when people talk about FRP, they mean merely "inspired by FRP" ** Naming is hard - should really be talking about "Compositional Event Systems" - Conal Elliott tweet - http://bit.ly/rx-commit - http://bit.ly/reactive-cocoa-commit ** what's the difference? - every construct in FRP has a precise mathematical definition - free of side-effects - kind of like Haskell's IO monad ** history - 1997: created in haskell - other haskell libs - reactive-banana, netwire, sodium - FRP-insired: - Rx[.NET/Java/JS], baconjs, reagi (cljs) - main abstractions: Behaviours and Events - traditionally: #+begin_src haskell type Behavior a = [Time] -> [a] type Event a = [Time] -> [Maybe a] #+end_src - this talk: compositional event systems ** motivating example - imperative code to iterate over a list - lots of changing state - functional code - we describe what, but not how - no mutating variables - gain reusable single-purpose functions - CES has similar principles - think of key presses as a list of keys over time - http://bit.ly/rxjava-github - http://bit.ly/rxjs-github - subscribe to event sources, filter/transform them - map behaviour to event streams - say, by sampling every second - ~flatMap~ / ~selectMany~ ** network IO - rather than events from keyboard, mouse etc - in javascript: callback hell :( - on jvm: clojure promises don't compose - promises in js are slightly better but have limitations ** demo: simple polling app - partition/zip ** quote - "FRP is about handling time-varying values like they were regular values" ** why not core.async? - core.async feels like it's a lower level of abstraction - it's a great foundation for an FRP-inspired framework - reagi is built on top of core.async ( http://bit.ly/reagi ) ** bonus example: reactive API to AWS - retrieve list of resources from a stack - for each EC2 instance, get status - same for each RDS instance ** Q: when you do you jump from handling manually to observables? - my rule of thumb is if I need anything more than a single callback, I'll use this (or core.async) ** Q: have you used RxJava from clj? How nice is it? - works great, so does RxJs * Stuart Sierra, Components: Just enough structure ** architecture - software architecture is very simple(!) - presentation - business logic - DB - actually, much more complex - config - connections to external resources - monitoring - queues - sessions/connections in pools - process state - thread pools - caches - schedulers ** Java: structure built in ** clojure: not much structure - clojure namespaces aren't classes - they're not instantiable - ~def~ creates a singleton - ~(def foo (atom ..))~ creates global mutable state - bootstrapping #+begin_src clojure (defn start-all! [] (database/connect!) (create-queues!) (start-thread-pool!) .... (start-web-server!)) #+end_src ** component - immutable data structure (map or record) - public api - management lifecycle - relationships to other components - It's an object (ssh!) - not using it to represent data ** State wrapper component - ~(defrecord DB [host conn] ....)~ - opaque to most consumers (by convention) ** Public API - fns take component as an argument ** Lifecycle: Constructor - set up initial state - /no side effects/ ** Lifecycle: Transitions - side effects happen here: #+begin_src clojure (defprotocol Lifecycle (start [component]) (stop [component]])) #+end_src - ~start~ and ~stop~ return an updated version of the component ** Service provider component - ~(defrecord Email [endpoint api-key ...] ...)~ ** Domain model? - traditionally intermingle data and behaviour: #+begin_src java public class Customer { private String name; private Address address; public void notify() {...} //... } #+end_src - Let data be data - just use a map ** domain model component - represent aggregate operations - ~(defrecord Customers [db email])~ - db and email are other components, used by the customer component - entirely interacts through their public APIs - to construct a Customers instance, need to get its dependencies ** system map - takes created but unstarted components: #+begin_src clojure (defn system [...] (component/system-map :customers (customers) :db (db ...) :email (email...))) #+end_src - to start the system, understands dependencies and works out correct dependency order to start each component - then wires each component up to the correct (started) dependency - stopping the system is similar but in reverse dependency order - Before start, dependencies not filled in yet (just nil) - after start, fill in dependencies - the system is just a map - so if I want to inject a test stub, I can just assoc it in: #+begin_src clojure (defn test-system [...] (assoc (system ...) :email (stub-email) :db (stub-db))) #+end_src - works as long as I do it before starting any services ** DB for testing - fixtures to inject into database - mocking the db is too hard unless you use datomic 😏 ** var substitution & asynchrony - with-redefs and binding are delimited in time - problems if you dispatch to another thread - potential race conditions - tightly coupled to implementation - wrong level of granularity ** Entry point: main - exactly one mutable global for the whole system - ~(def sys (atom nil))~ - use ~reset!~ not ~swap!~ because start and stop are side-effecting and swap! might call multiple times - (@samaaron ed: uses agents for this sort of thing) ** Web app: static routes - defroutes considered harmful ** renaming dependencies - you can merge systems - name common components with shared keys: #+begin_src clojure {:a/web-app .. :a/server ... :db ... :email ...} {:b/web-app .. :b/server ... :db ... :email ...} (merge system-a system-b) #+end_src ** core.async - components take channels as state - decouples components from one another - system creation can create the channels you want and wire them up ** summary - advantages - once you're used to the patterns of clear dependencies and boundaries, you maybe don't even need the library anymore - isolation, decoupling - testing, refactoring - automatic ordering of start/stop - easy to swap in alternate implementations - everything at most one map lookup away - disadvantages - requires whole-app buy-in - won't get a lot of the benefits without this - porting an existing system can be tedious - system map is too big to inspect visually - cannot start/stop only part of a system - may try to fix someday but don't really understand how yet - possible future - "init" acquires resources but doesn't start? - "close"/"stop" separation -- close acquired resources and discard dependencies so they can be GC'd - (the "stop" method doesn't dissoc anything) - dissoc stops a record being a record - you might want to use that state again - handle mutable containers for systems - currently, library code doesn't care -- you can use an atom or a var or whatever - allow individual components to start, stop, or change at runtime - deref container and get "current" component with latest deps - catch errors, mark component as "failed" - this is the tricky part * Philip Potter, Generative testing with clojure.test.check - sorry, I was too busy to take notes for this for some reason - preliminary notes: - [[https://gist.github.com/davegolland/3bc4277fe109e7b11770][davegolland/generative-schema.clj]] -- demonstrates how to convert schemas to test.check generators - (post-hoc addition: [[https://github.com/miner/herbert#testcheck-integration][herbert]] is like schema and comes with test.check integration; thanks to @minimal for pointing this out) - slides: http://www.philandstuff.com/slides/2014/euroclojure.html * Chris Ford, the hitchhiker's guide to the Curry-Howard correspondence - number of papers published today by the foremost expert on the Curry-Howard correspondance... - 1 - Don't panic! - Gödel's incompleteness theorem ** introduction - a → a - this is a proposition in logic - but it's also a type - the type of the identity function ** our heroes - Haskell Curry - 1958: textbook on combinatorial logic - didn't necessarily understand how revolutionary this idea was - William A. Howard (1969) - not only does a type correspond to a proposition, but: - a function with a type corresponds to a proof of a proposition - "The formulae-as-types notion of construction" - finally published in 1980 ** modus ponens - (a → b) → a → b - modus ponens - type of apply (haskell or idris): #+begin_src haskell apply :: (a -> b) -> a -> b apply f x = f x #+end_src - the implementation here corresponds to a proof of modus ponens - apply works with any types a and b - modus ponens works with any propositions a and b - view the type "Integer" as the proposition that integers exist - any example -- say, 65 -- counts as a proof of this proposition - here, we use ~(3==)~ to prove that ~Integer -> Bool~ is populated #+begin_src haskell (3==) :: Integer -> Bool apply (3==) 4 False: Bool #+end_src ** composition - (a → b) → (b → c) → (a → c) - type of function composition - ~length : List a -> Integer~ - ~(3==) : Integer -> Bool~ - ~comp length (3==) : List a -> Bool~ - if we accept that ~List a~ exists, we now prove that ~Bool~ exists ** axioms? - a → a - (a → b) → a → b - a → b → (a,b) - if I can build an ~a~, and I can build a ~b~, then I can build an ~(a,b)~ - (a,b) → b ** bottom type - a → b - (a → b) → (b → a) - neither of these are true in general - the bottom type: ⊥ is guaranteed to have nothing in it - represents falsity in the Curry-Howard correspondance - represents something it's impossible to prove, because it's not true #+begin_src idris AnythingGoes : Type AnythingGoes = (a : Type) -> a cantProveItAll : AnythingGoes -> _|_ cantProveItAll f = f _|_ #+end_src - cantProveItAll shows that AnythingGoes is uninhabited (because if it weren't, it would imply that ⊥ was inhabited) ** harmless(?) - types prove our program correct? - types only get us so far - can still get runtime errors if the types check out ** mostly harmless. - are types defective? - haskell will crash at runtime despite an advanced type system - ~head []~ isn't defined ** enter Idris - Edwin Brady, creator of Idris - (/and whitespace/) - killer feature of Idris: - allows you to make condescending remarks about the Haskell type system - although it's really a dialect of Haskell - example: #+begin_src idris Type Nat = Z or (S Nat) Type List = [] or (x :: List) data Vect : Nat -> Type -> Type where Nil : Vect Z a (::) : (x : a) -> (xs : Vect n a) -> Vect (S n) a #+end_src - in Haskell, types can be parameterized on other types - in Idris, they can also be parameterized on *values* as well as types - ~Vect 2 Integer~ is the type of vectors which contain exactly 2 Integers - or rather, ~Vect (S (S Z)) Integer~ #+begin_src idris head : Vect (S n) a -> a head (x::_) = x head [] Can't unify Vect 0 a with Vect (S n) iType #+end_src - trying to take the head of an empty vector is a *compile-time* error ** concatenation - signature: ~Vect m a -> Vect n a -> Vect (m+n) a~ - sort: ~Ord a => Vect m a -> Vect m a~ - would have caught Phil's ~my-sort~ which dropped duplicates(!) - didn't manage to get this implemented in the lunch break - it's not theorems for free 😉 ** even number family of types #+begin_src idris data Even : Nat -> Type where Zero : Even Z Next : Even n -> Even (S (S n)) Zero : Even Z Next (Next (Next Zero)) : Even 6 #+end_src - can now show that even numbers sum to even numbers: #+begin_src idris add : Even m -> Even n -> Even (m + n) add Zero y = y add (Next x) y = Next (add x y) #+end_src - Although we've really proved that: - there exists an operation which takes ~Even n~ and ~Even m~ and returns ~Even (n+m)~ - we chose ~add~ but could have chosen any other name - can prove that 42 is even - ~Even 3~ is a valid type - ~Even 3 -> _|_~ - proof in slides ** the unit type - ~()~ represents truth - you don't need anything else to prove this - you can construct it without context - (you could use ~Even 42~ to represent truth too) - so ~LifeTheUniverseAndEverything -> Even 42~ :) ** References - http://idris-lang.org - http://brianmckenna.org - http://wadler.blogspot.co.uk ** Q: is there a way to specify that sort's return value is sorted? - @bodil thinks it's true :) ** Q: is the type-checker guaranteed to terminate? - it's equivalent to the halting problem * Anna Pawlicka, Reactive data visualisations with Om - slides: http://www.slideshare.net/annapawlicka/reactive-data-visualisations-with-om - code: https://github.com/apawlicka/om-data-vis - @AnnaPawlicka - Data Engineer, Mastodon C - "Big Data" startup - (or Big "Data startup"?) - "I work for Bruce" - it's been fantastic learning Clojure over the last years ** Technologies *** D3 (data-driven documents) - to visualise data - table of numbers, bar chart, whatever - data bound to DOM - interactive - transformations driven by data - huge community - huge number of plugins and extensions - Higher level libs available - hide the complexity of d3 - but if you need to tweak the underlying d3 it's still available *** leaflet.js - layer on top of d3 - mapping data - tile layers, vector layers - user interaction *** dimple.js - charting library on d3 - bar charts *** react - (interface components) - solves one problem: complex UI rendering - just the V of MVC - say no to "two-way data binding" - re-renders the entire UI - sounds like a bad idea - actually quite performant, due to: - virtual DOM - diffs between previous and next renders of a UI - less code - shorter update times **** react lifecycle - IInitState → - IWillMount → - IShouldUpdate → - IRenderState - IRender - Om handles most of these for us (particularly IShouldUpdate) *** Om - entire state of the UI in single piece of data - immutable data structures = reference equality check - ~shouldComponentUpdate()~ can be overridden to take advantage of this - snapshottable, free undo *** Liberator & core.async - component interaction - liberator: RESTful apis via ~defresource~ macro - core.async - js developers would freak out looking at it - get blocking calls without browser freezing ** data sources - for example, local sensors - may want to perform some sql queries to see patterns in your data - may wish others to retrieve data through API (if they don't like our chart) ** Chart & API demo - user interacts, triggers API calls to fetch data, updates graph in real time - chart implementation - om/IInitState to construct - om/IRenderState - to update - device-form - om/IWillMount to read shared info to find API endpoint - om/IRenderState - form-row - chart-figure *** last.fm chart - chart based on last.fm playlist - input box takes username, calls last.fm to find that user's playlist - chart then shows bands from most popular downwards ** interactive maps - input box for postcode lookup - uses google geocoding api to get coords - can click map to create marker & display coordinates - app-model stores map location and coordinates panel contents - nice use of core.async ~sliding-buffer~ 😎 - ~(go (while true ...))~ could be ~(go-loop ...)~ ? dunno ** summary - fast rendering and interactivity is yours with js + cljs + om - immutability = efficiency - sane application structure - (philandstuff ed: this presentation is very visual, just watch the video!) * algernon, The Face of Inspiration, or how Clojure helps bring Lisp to Python - github: algernon - twitter: algernoone - sorry, I was in the hallway track for this 😦 * Malcolm Sparks, Assembling secure clojure applications from re-usable parts - @malcolmsparks, juxt ** warning! research, evolving ideas, alpha quality ** juxt/modular - set of components compatible with stuartsierra/component - http-kit, bidi router, mustache templating ** juxt/cylon - security components - login form - session - user domain - hashing - authn and authz ** assertion - libraries are great - systems are complex ** assertion 2 - a meta-architecture, that can scale to hundreds of diverse projects, is useful ** architecture - don't want to just port Spring MVC to clojure - components, dependencies, protocols *** components - reusable bits *** dependencies - wiring of components together - since the system is in a var, I can do a tree-walk on the system, and show it - can visualise it with dagra and svg rendering, and react - slide deck which shows its own wiring - I'm So Meta, Even This Acronym *** protocols - integration surface - necessary for component interchangeability - example: bayonet light-bulb fitting - can plug a light bulb into it - light bulb dies -- replace it! - over time, can replace entire system by replacing parts - hidden couplings - copied code - database schemas and sql queries - URI formation & URI dispatch - have to change things in multiple places to effect change - juxt/bidi - dispatch and forge URIs from the same route data ** component example - constructor - defaults with merge - schema/validate - components are units of cohesion - implements multiple protocols: - component/Lifecycle - WebService - JavaScripts - TemplateData ** intermission - maze creation in cljs - "drunken walk" algorithm - the first time you visit a space, you break down the wall - core.async visualisation and demo of ~map<~ ** the index pattern ** the interceptor pattern - a component is wired in between two components ** shared dependency pattern ** security *** challenge - don't want to re-implement security components - tried-and-tested security by default - flexibility of 'roll-your-own' *** example: website - router routing between sub-websites A and B - add a login form to the router, which uses: - user-domain - password-algo (eg scrypt) - user-store (eg cassandra) - session-store - all comes from cylon - add authorization component to website B (again from cylon), using: - authenticator - session-store (same dep as above) ** summary - http://modularity.org - google group: modularity ** demo - ~lein new modular myapp~ - ~lein new modular myapp +cljs~ - ~lein new modular myapp +cljs +security~ - ~lein new modular myapp +cljs +security +devtools~ - ~(dev)~ fn - if your code doesn't compile on your repl, then you just get loads of stack traces - secure content, rather than URI routes - there may be multiple routes to the content - ~restrict-handler~ to wrap a response in a RestrictedHandler which implements IFn to look like a fn and make it invokable ** Q: what about hypermedia to decouple URI dispatch & formation? - we don't have HATEOAS because it's quite hard - want to make it easier * that's all folks! - thanks for reading :)