Several approaches have been developed in the domain of EUP. The most popular approach is the Spreadsheets which are widely applied by individuals and businesses (e. g. grading by teachers, spreadsheets for accountants etc.). Scripting Languages is another technique which, in the context of EUP enables the extension and adaptation of existing applications by end‐users. Scripts are the most powerful EUP tools, however they present end‐users with considerable learning cost and they are prone to errors. Another approach which is based on scripting languages is the Natural Languages phrases interpretation. While the latter sounds promising for programming purposes, existing approaches seem to have problems and restrictions in use; that’s why they are applied only in research and not in industry. The aforementioned approaches are based on text‐based environments and it seems that they aren’t the best choice for portable EUP applications for smart objects. However, there is another approach called Visual Programming which is based on graphic artifacts that have been mapped to the correspondent high level functionalities according to each Visual Programming Language (VPL). The most popular category of such artifacts for the VPLs are based on jigsaws which first appeared in Scratch [1]. Another approach of VPLs is flow diagrams; for example Microsoft VPL [2] is based on flow diagrams for building robotics applications.
In our work, we develop a portable touch‐based visual end‐user programming environment for personal applications, running on typical smart phones and tablets, where end‐users may discover smart objects in their own surroundings and compose custom applications reflecting their own needs. Similar to our approach is HomeKit [3] which is a framework for communicating and controlling connected accessories in user’s home. Also, the graphical control logic editor oBeliX [4] is a proof of concept tool for IoTSyS [5], an integrated middleware of IoT.
The rest of this chapter is organized as follows; In Sect. 71.2, existing smart objects in people’s daily life are described. In Sect. 71.3, we discuss potential scenarios that end‐users may develop. In Sect. 71.4 our approach in visual programming for smart object applications is described. Then, we present a case study developing the scenario described in Sect. 71.3.2. We continue by discussing the potential of digital markets for smart object applications. Finally, we briefly discuss the visual programming requirements of end‐users in the context of IoT.
71.2 Smart Objects in Daily Life
The main building blocks of the IoT are smart objects. The concept of the IoT started in early 2000’s with the RFID tags, and network connectivity of physical things. Furthermore, there are numerous smart devices available on the market which are already used in the context of IoT. In particular, devices which are commonly used in daily life have been evolved to smart connected devices by offering extra services and automations (e. g. tracking information, remote control, exchanging data with other smart objects etc.). The refrigerator is a representative example of a device used on a daily basis. Its main function is to maintain and store food items and fresh produce. But as a smart object, apart from the above functions, it will also be able to do other more complex functions such as identifying, enumerating, and holding important information about the food items it contains. Smart refrigerator notifies users when a food item is close to expire or if it has already expired. Furthermore, the refrigerator is able to display through an embedded screen, recipes based on the food items that are currently stored. Moreover, the users can remotely view what is stored in their refrigerator.
In addition, apart from the physical connected things and the smart devices, there is a huge number of applications online and day by day this exponentially increases. These applications could be used in the world of IoT and could be considered as smart objects which are connected online and are able to communicate through web‐services. Such applications could be available via digital marketplaces. Examples of applications could be weather forecast, a clock, a chronometer etc. Furthermore, examples of such applications that could be interoperated with the smart refrigerator are a nutrition calendar and online shopping. Using these smart objects, the user will be able to program a weekly meal plan based on which the refrigerator could automatically place online orders in authorized food shops.
Taking into account the aforementioned about the smart objects which are available in people’s daily life, people may like to have custom automations based on their needs. In the next section, we discuss such example applications.
71.3 Scenarios
Using existing smart objects, we discuss potential scenarios which could be developed by end‐users based on the visual end‐user environment we develop. However, the scenarios discussed below are just indicative, since by offering end‐user programming features and due to the fact that there is a huge variety of smart objects available, the possibilities are endless.
71.3.1 Remote Hospitality
Many times people would like to be at home (or office) when their doorbell rings but instead they happen to be somewhere else. This happens either when there is a meeting for which they couldn’t be there on time or in case of a surprise visit. Before the existence of IoT concept, visitors could only call the potential hosts in order to communicate with them. Thanks to IoT, people are able to use smart doorbells which are supported by appropriate software applications. The latter notify users when the doorbell rings and help them communicate with the person who rang it. On the one hand, smart doorbell software provides support for all possible services of the device, on the other hand, it is impossible to provide support for other smart objects that users would like to use with the smart doorbell. For example, end‐users may like to have an application which uses home smart objects in order to host visitors remotely until they go back at home as depicted in Fig. 71.1. The smart door gives access to the visitors, while the smart lights turn on or window blinds open depending on the time of the day. Furthermore, the home temperature can be regulated using the air conditioning system and the thermometer. Then, the smart Hi‐Fi or TV could take on the visitors’ entertainment. In addition, drinks can be prepared by the smart coffee machine or the smart kettle.
Fig. 71.1The flow of Remote Hospitality Application and the involved smart objects
Someone would wonder why we have to create a new application using smart objects and not use all the provided applications from our smart objects. The answer is twofold. Firstly, users would like to have custom automations without having to use each of the applications of the smart objects. In addition, running applications for each smart object would be impossible in case of using several smart objects for one task something which would be a common scenario in the concept of the IoT which is based on the pervasive deployment of smart objects around the world. Secondly. there are several cases that smart objects are based on the events and data of other smart objects. A representative example of such application is discussed on the next section describing morning automations.
71.3.2 Morning Automations
One of the most difficult times of the day for people is waking up and their morning habitual tasks. There are several things that people have to do when they wake up such as, have a bath, prepare their breakfast, be informed about the news and their messages, prepare for their work, leave home for work etc. Using the existing smart objects, several processes could be automated and users would gain some more minutes of sleep, find their home temperature regulated, not forget to be informed about the news, leave home without worrying if they forgot to lock the windows or turn off lights, electric devices etc. All these automations can be accomplished when related events are triggered as depicted on the Fig. 71.2.
Fig. 71.2Morning Automations triggered by environment events
The first event of application is based on the time that the alarm clock is programmed to ring. When the event is fired, the alarm clock is switch
ed off before it rings, then the air conditioning regulates the home temperature, while heater starts preparing water for a morning bath and the coffee machine prepares the first coffee of the day. Then, when water for the bath is ready, the alarm clock rings and the window blinds open. Also, when coffee is prepared, the coffee machine notifies the user. Afterwards, when the user opens the bath door, the smart Hi‐Fi automatically starts playing music and the smart bed makes itself. Afterwards, when the user starts serving coffee (once she has finished with her bath) music stops and it is time to catch up with the news and view the daily tasks she has to do, messages or email she has received. Finally, when leaving home for work, smart objects take on the home safety by locking all windows, window blinds and outdoors which are still open, switching off not used electric devices such as the air conditioning, the TV etc. turning off the lights and activating the alarm system.
71.3.3 Self‐Caring Home
Continuing the previous scenario of morning automations, end‐users could design automations for tasks which are required by their home such as cleaning, gardening, shopping etc. using appropriate smart objects. However, these are mainly calendar based tasks of the home that are executed repeatedly either with the specific frequency or not. The events defined for the application of home self‐care automations are categorized on three categories home cleaning, shopping goods and care for the garden as depicted in Fig. 71.3.
Fig. 71.3Home care automations triggered by smart objects and calendar events
The first task is programmed to be executed every day, when home is empty the windows open for half an hour for airing purposes. Then, windows close and the vacuum starts sweeping the house and then the house is mopped by the smart mop. In addition, when washing machine is full, it checks that it will not rain for the next 3 days and starts washing. When it finishes, the washing machine sends the respective notification. Also, the refrigerator checks for the availability of goods based on the nutrition calendar and in case something is missing it makes an order of goods. Additionally, tasks for care of the garden are defined as seen in the last part of Fig. 71.3. The first task cares for the watering of the garden by checking if the garden needs watering every two days and completes the task after checking that it will not rain for the next day. The second task cares for the fertilization and mowing of the garden by repeating the process every 12 days. In addition, before it starts the process, it checks that it won’t rain for the next two hours.
71.4 Visual Programming
71.4.1 Discovery of Proximate Smart Objects
The first task end‐users have to do for their development process is to discover and register the smart objects which will participate in their applications. Furthermore, as discussed previously in the context of IoT, there are numerous proximate smart objects available in people’s surroundings. Hence, the management of the smart objects through the visual end‐user programming platform is necessary. The smart objects may be positioned in different locations where people operate such as the home, the workplace, the gym etc. and smart objects’ position may change (e. g. smart watch). Based on this fact, our approach in order to facilitate the smart objects organization, is to be able to define virtual environments in which the smart objects could be placed during the registration of the platform by selecting the respective environment. Furthermore, inner environments can be defined in each environment (e. g. Home, Home/Living room, Home/Bedroom, Home/Garden).
In addition, during the registration of the smart objects, the end‐user is able to insert respective icons improving the visualization of smart objects. An example of environments and smart objects is represented in the left of Fig. 71.4, using an example of two defined environments the home and the office where smart objects have been placed. Also, the registration of a light smart object in the home environment is presented on the right screen of Fig. 71.4. Moreover, end‐users are able to hide functionalities that may not like to use in their development process.
Fig. 71.4View of defined environments of smart objects (a). Registration process of a light smart object (b)
Furthermore, using the concept of environments, the end‐user can have a quick wide view of the smart objects’ information in each environment and can more easily focus in the smart objects of a specific environment in order to develop automations for this environment. The latter is common practice, and it doesn’t restrict the end‐users from developing custom automations which include smart objects from more than one environments. An example of such automations could be a “back to home” application using smart objects exist both in the office and at home environments. Moreover, the development process does not require that the smart objects should be connected thanks to the registration of the smart objects in the platform which has generated and maintained the smart object specifications.
71.4.2 Visual Programming of Smart Objects
Using the VPL of IoT, the end‐users main task will be the handling of smart objects that will be involved in their applications. Each of the smart objects provides different functionalities which have to be published via a well‐defined API in the context of IoT. Based on this, our system builds the smart object data to a JavaScript object from which the respective visual programming elements are generated and revealed to the visual programming environment.
Using as example the air‐conditioning, total kind of smart object functionalities are depicted in Fig. 71.5 by presenting the mapping of the source code in JavaScript and the visual code of our visual programming language. There are four main categories identified for the smart object functionalities which are based on their I/O. In particular, there are functionalities without I/O such as the action “turnOn” of the air‐condition which is depicted in Fig. 71.5. The visual element of our VPL is the blue box which consists of the respective smart object icon and the inner white box includes the selected functionality. By clicking the white box area, the end‐user can view total functionalities of the air‐condition and change the selection. The mapping of the generated source code of this element is the statement “AirCondition.turnOn();”. Furthermore, there are functionalities with input or output which are visually represented in Fig. 71.5. Based on the types of the input arguments and the output when white boxes are clicked, the visual programming editor gives only the valid possible variables, values etc. in order to facilitate the end‐users by limiting or eliminating their errors. Finally, the last category of smart object functionalities based on I/O includes both input and output elements.
Fig. 71.5Air‐Conditioning object functionality (left), Use of the object and mapping to the visual programming language (right)
Moreover, another category of smart object functionality is based on the responses which are triggered asynchronously and one or more actions could be defined for execution when it responds. The air‐condition supports an asynchronous response “onChangeTemperature” as depicted in Fig. 71.5. The source code actions which will be executed in case of the response corresponds to a callback function in JavaScript while this maps to a block of statements which are the respective statements of the callback function body as shown.
Furthermore, the main context of the applications in the IoT is focused on the ecosystem of smart objects that exchange data and interact in an automatic or semi‐automatic way based on events. As result, the design of events has to be one of the main concepts of the VPL for smart objects. Such events are the aforementioned asynchronous responses; however, these events are not adequate to define total events of applications in the context of IoT efficiently. Firstly, there are smart objects which partially support or not support functionality of asynchronous responses. In this case, the VPL has to support these events by defining such events and core takes on the functionality by using poll
ing which checks the concerned value with appropriate frequency. Moreover, end‐user would like to define event actions when combined data of more than one smart objects values change state. In addition, there is the need to define calendar events like the presented in the scenario of self‐caring home. These events are separated in three different types based on the times to be executed. First category is the non‐repeatable events which are executed once (e. g. on Tuesday smart garden fertilizes the grass). Second category is the finite number of times that events will be executed (e. g. On Tuesday, Friday smart garden sprays the plants suffer from a disease). The last category is the periodic execution of events (e. g. Every 2 days the grass is watered). Also, in case of IoT there are events which could be connected or disconnected to the application system by other events and vice versa. In the following case study, some of the VPL event elements have been used as shown in Fig. 71.7.
71.5 Case Study
Using the visual programming tools we develop, we have carried out a case study based on the morning automations scenario described in Sect. 71.3.2. The smart objects are included in morning automations are shown in Table 71.1. Table 71.1Morning Automations Smart Objects and their functionalities
Smart Object
Functionality
Events
Alarm Clock
Switches on/off
Digital Marketplaces Unleashed Page 109