2D/3D Web Visualization on Mobile Devices Yi Wang12 , Li-Zhu Zhou1 , Jian-Hua Feng1 , Lei Xie3 , and Chun Yuan2 1

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China. 2 Institute of Information Technology, Graduate School at Shenzhen, Tsinghua University, Shenzhen 518055, GuangDong Province, China. 3 School of Creative Media, City University of Hong Kong, Tatchi Avenue, Kowloon, Hong Kong, China. [email protected] ; [email protected] ; [email protected] ; [email protected] ; [email protected]

Abstract. Visualization is able to make the result of Web search and Web mining more intuitive and make the search/mining more productive. However, the technical limitations of mobile devices make it difficult to port visualization methods from desktop computers to mobile devices. In this paper, we present what we learned on engineering 3D Web visualization on both high-end and low-end mobile devices as the MWeb3D framework, which forms a distributed pipeline that move intensive computation from the mobile devices to server systems. Some important issues of this strategy includes: (1) separating visualization from graphics rendering, (2) encoding visual presentation for transmitting via bandwidth-limited wireless connections, (3) user interaction on mobile devices, and (4) highly efficient graphics rendering on the mobile devices. We will show fruitful experiments on both PDA and mobile phone with photos taken from both simulator and real mobile devices.

1

Introduction

After decades of development, the Web services provided all over the Internet has become a major source of information. Researches on Web information retrieval techniques, including Web searching and Web mining, help the users to select closely related information. Researches on Web visualization techniques are aimed at presenting the retrieved information intuitively and comprehensively [1] [2] [3] [4] [5]. Visualization helps people to understand information intuitively and solve problems effectively. In addition, visualization techniques are efficient on presenting rich information in limited display area. If we can adapt 3D Web visualization techniques to mobile devices, like PDAs and mobile phones, users would be able to exploit the fruitful Web information to help solving problems productively anytime and anywhere. Unfortunately, technical limitations of the mobile devices make it very difficult to port existed Web visualization techniques from desktop systems to mobile environment [6].

From the perspective of 3D Web visualization, the major technical limitations of the mobile devices include: – Small display with low resolution and a few colors. In contrast, most existed Web visualization techniques were designed for desktop systems with large screen and millions of colors. – Limited computing capability, such as CPUs without floating-point units, memory bus with limited bandwidth, and the lack of graphics hardware. – Slow wireless connections, such as Bluetooth, which can not afford the traditional way of Web visualization that retrieve large amount of Web information to the mobile device before analysis and visualization. For some specific visualization tasks, such as the 2D map visualization [6], the above difficulties can be addressed by modifying or designing visual presentations especially for mobile devices. However, a more general solution, especially for 3D visualization, requires systematic strategies that are applicable to various visualization tasks on mobile devices.

Fig. 1. The visualization pipeline of the MWeb3D framework.

In this paper, we present our exploration of 2D/3D Web visualization engineering on both high-end and low-end mobile devices as a general solution, the MWeb3D framework, which features the following advantages: – Distributed visualization pipeline. To release the mobile devices from the intensive computations, we separate traditional meaning of “visualization” into two parts: the generating 2D/3D visual presentations and the 3D graphics rendering. By moving the former part to MWeb3D server systems, we organize all computational components as a distributed pipeline, as shown in Figure 1. Only the last stage of the pipeline, graphics rendering and displaying, is performed on mobile devices. – Encoding and transmitting visual presentations. Because the visual representation of the Web information generated by the MWeb3D server have to be encoded and transmitted to the mobile device for rendering and displaying, and because the limited wireless bandwidth, we discuss the designing of encoding schemas that can encode the visual presentations compactly. – User interaction on mobile devices. Interactive visualization on mobile devices, especially the navigation in visual presentation, has to adapt to the

small screen size and special input methods, such as the stylus of PDAs and numeric keypad of mobile phones. We discuss adapting interaction methods, such as the level-of-detail (LOD), to mobile devices and incorporating with the mobile input methods. – Graphics computing on mobile devices. As graphics computing is wellknown of its computational complexity, we discuss properties of the computational facilities on mobile devices and how to make full use of them to achieve efficient graphics rendering on mobile devices.

2

The Visualization Pipeline

Visualizing Web information includes several major computational tasks: (1) retrieving Web information via Web searching or Web mining; (2) analyzing the retrieved information and generate the visual representation of the information; (3) rendering the visual presentations into image; and (4) displaying the image onto screen. Traditional information visualization approaches, designed for desktop systems with promising computational power, usually implement these tasks monolithically as a single piece of software running on a single computer. However, for mobile devices, any of these tasks may be over-challenging. Therefore, in the MWeb3D framework, we distribute these computational tasks to different computers, in particular, we move the former two tasks on powerful sever systems and leave the latter two on the mobile devices. The distributed tasks are organized as a pipeline, as shown in Figure 1, where, task (1) runs on the Web search/mining engine sever, such as Google or the VisMap server [5], task (2) runs on the MWeb3D server, and tasks (3) and (4) run on the mobile client. 2.1

Discussion on Separating Visualization from Rendering

It is arguable that not all visualization methods can have the rendering stage separated from the generating of visual presentation. However, because the Web searching/mining is in fact a process that abstract knowledge domain [7] from Web information, visualization of Web searching/mining result is intrinsically a process that represents abstracted concepts or entities of the knowledge domain with graphics elements and organizes the layout of these elements to represent relationships among the concepts or entities, as the examples shown in Figure 3, 4 and 5. So, it is natural to encode the layout and transmit to mobile devices for rendering and displaying. Another organization of the pipeline that even lightens the computational intensity of the mobile devices is to move also the task (3) from the mobile device to the MWeb3D server system. But this organization requires transmitting the rendering results as images or video to the mobile device, which is far from affordable by the limited bandwidth supported by the mobile devices, such as Bluetooth.

However, running task (2) as a separate stage in the pipeline is promising, because this step usually have to do layout with iterative optimization algorithms, such as the generation of treemap [8] [9], of cluster map [10] and of pathfinder networks [7], that are computationally intensive. 2.2

Serving Multiple Simultaneous Clients

(a)

(b)

(c)

(d)

Fig. 2. Experimental devices used to form the distributed rendering pipeline. (a) The MWeb3D server running on a cluster of 4 Linux-based computers. (b) The Bluetooth adapter that connects each computer in the MWeb3D server cluster with the mobile devices. (c) A Nokia N70 mobile phone that renders and displays a treemap visual presentation. (d) An HP iPAQ PDA that runs the same treemap visual presentation.

Because the MWeb3D server usually have to generate visual presentations for multiple mobile clients simultaneously, to cope with the possibly large number of simultaneous clients, we implement the MWeb3D server as a Linux cluster (c.f. Figure2(a)) using OpenMOSIX, which enhances the Linux kernel with support to distribute multiple tasks over computers in a local area network. Once a mobile client connects to the MWeb3D server cluster, the OpenMOSIX is in charge of allocating a certain computer to serve the client. In the MWeb3D framework, the basic serving unit for an MWeb3D server is called an MWeb3D transaction. A complete MWeb3D transaction is composed of the following steps: 1. the mobile client requests connection and passing the search request; 2. one computer in the MWeb3D cluster is allocated to serve the client and rely the search request to Web searching/mining engine; 3. after accepts retrieved Web information, the responsible MWeb3D computer generates and encodes the visual presentation, 4. after the visual presentation is transferred to the mobile device, the client program renders and displays the presentation.

Because the MWeb3D transaction is a pure read operation, maintaining its consistency is easier than maintaining the transaction of database management systems. In our experimental system, when either side of the Web searching/mining engine, MWeb3D server and the mobile client times out, the whole transaction is aborted. This helps ensuring the high responsibility of the MWeb3D system. We measure the computational intensity of a MWeb3D server computer in the cluster by the number of MWeb3D transactions that the computer is processing. By constraining the maximum number of simultaneous transactions that each computer allows and allocating the most disengaged computer to serve new transactions, we implement a simply and fast tasking balancing algorithm, which makes that the overall processing capacity of the MWeb3D cluster grows linear to the number of computers in the cluster.

3

Compact Encoding of Visualization Result

concept user activity social relationships

graphical primitives spheres cylinders on spheres edges between spheres

properties user-id, position, color user-id, length user-id-1, user-id-2

Fig. 3. Illustration of the pathfinder network and corresponding visual schema.

Because we distributed the task of generating visual presentation and the one of graphics rendering on the MWeb3D server and the mobile device, the generated visual presentation have to be encoded compactly and be passed through the limited wireless bandwidth to the mobile device for rendering. Because, generally, any visual presentation can be represented by a layout of graphics primitives, such as points, facets, and curves, we can design a general encoding schema that supports these primitives. Such a general schema will concise, because it only

need to support a few types of graphics primitives. However, if we encode according to such schema, the encoding result may be lengthy. Take the 3D visual presentation shown in Figure 3 as an example: This visual presentation is called pathfinder network [7], it describes the “virtual social relationship” between a group of users of a Web forum [5], where, each small sphere represents a user, the vertical cylinder on the sphere represents the number of posters that the user had submitted to the Web forum, and the edges that connects the spheres represent the intimate social relationships between pairs of users. These intimate relationships are mined by counting the frequency that two users post to the same topic. If we encode this visual presentation by the “general schema”, we will have to describe the whole scene by a huge number of 3D triangular facets. Such encoding result is far from affordable by contemporary wireless connections such as Bluetooth and even Wi-Fi. If we define the encoding schema specifically for pathfinder networks, we can encode the visual presentation as the 3D position of the spheres (users), the length of vertical cylinders (the number of posters), and the connections between the users (the edges), as shown by the table in Figure 3. Although this new definition of encoding schema is specific to pathfinder networks, it brings in compact encoding result. Specificity is one of the many approaches to incorporate semantics, or domain knowledge, into the encoding schema. When more domain knowledge is implicitly incorporated, the less information would have to be encoded. Moreover, incorporating semantics also makes customization easier. For example, the specific schema designed for the pathfinder network can allow mobile users to specify their favorite colors of the users, the vertical cylinders and the connections. On the opposite side, as can be seen from the above example, the specificity also complicates the encoding schema. In the balancing of complex/concise encoding schema between compact/lengthy encoding result, we choose to design encoding schema for each kind of visual presentation. Beside the schema specific to the pathfinder network, we also defined and implemented visual encoding schema for the treemap [8] [9] and the cluster map [10] as shown in Figure 4 and Figure 5. In Figure 4, the cluster map visualizes three discussion topics in a Web forum: UNIX, Solaris and Linux. Users are grouped as their participations to these topics. Each group are represented by a drop-shape, which contains a set of spheres that correspond to the users. In Figure 5, the treemap visualizes the hierarchy of discussion categories. Each category is represented by a rectangle, whose area is proportional to the popularity. Users that often post in certain category have their photos shown as icons in corresponding rectangle. The photos that show the visual presentation rendered and displayed on HP iPAQ PDA are shown aside to the illustrations, and the definition of visual encoding schema for these visual presentations are listed below the illustrations.

concept topic user-group user

graphical primitives properties crystal spheres with text label topic-id, text-label drop-shapes group-id, color, topic-id-list spheres in drop-shapes user-id, group-id

Fig. 4. Illustration of the cluster map and corresponding visual schema.

concept graphical primitives properties category rectangular area category-id, color, text-label, parent-category-id user icons with user’s photo or logo user-id, category-id, icon-image Fig. 5. Illustration of treemap and corresponding visual schema.

4

Interactive Visualization on Mobile Devices

Rendering and displaying the encoded visual presentation on mobile devices must adapt to the small screen and special input methods of the mobile devices, such as stylus of PDA and numeric keypad of mobile phone. Often, the small screen cannot hold all the details of the visual presentation, some previous works use interaction techniques to hide information that is currently out of users’ focus. In [6], the authors draw maps with hot-spot on the PDA. For each hot-spot that is out of the screen area, a red aura is drawn around the hot-spot and the boundary of the aura is extended to the current display. So the users can approximate the relative position of the hot-spot. In [11], the authors use fish-eye technique to draw a calendar consisting of a grid of daily arrangements. The currently focused day will occupy the most area of the screen while other days will shrink to small spaces. In our work, we propose to use the level-of-detail (LOD) technique to adapt the tree map and cluster map to mobile devices. For the tree map (c.f. back to Section 3 and Figure 5), which is designed to visualize a hierarchy of categories, we merge too small rectangles in deep layers into a single larger rectangle. Figure 6 compares a tree map being visualized in a window of a desktop system and in a window of PDA. The screen area of PDA is about 1/4 of the window area, so if the user wants to have a overall view on the treemap, many text labels and icons are hidden on the PDA screen. However, we can notice that the icons on the PDA screen have the same size as shown on the desktop system. When the user want to focus on a certain category that is represented by a rectangle, she/he can simply tap the rectangle with the stylus, and the tapped rectangle will be enlarged to occupy the whole screen. If the user want a step back to upper level, only a double-tap is required. Figure 7 shows different level of detail of the same cluster map (c.f. back to Section 3 and Figure 4) displayed on an iPAQ PDA and a Nokia N70 mobile phone. Because the screen size of Nokia N70 is about 1/5 of the iPAQ, details of the clustermap, including the spheres in the user groups are neglected on the mobile phone when the user is looking at an overview of the visual presentation. By pressing the keys 2, 6, 8 and 4 on the numeric keypad, users can switch among the topics (blue crystal spheres); by pressing 5, the user can enlarge the visual presentation to have a closer look to the selected topic and the user groups related with it. By pressing 0, the visual presentation is shrinked one level.

5

Graphics Computing on Mobile Devices

Graphics computing is one of the most challenging tasks, even for desktop systems and workstations. On most current mobile devices, there is no graphics acceleration hardware. Only a few top-end mobile devices have equipped with fixed-function graphics processors. This means that most graphics operations have to be executed by the CPU, or as usually said in the graphics society, “by software”. Unfortunately, most current mobile CPUs do not support floatingpoint computation, which, however, is heavily relied by graphics rendering. So

Fig. 6. Apply level-of-detail technique to rendering tree map.

Fig. 7. Apply level-of-detail technique on rendering cluster map.

developers have to use fixed-point techniques to do real number computation. What is potential to make use of is that, many mobile CPUs provide integer Single Instruction Multiple Data-stream (SIMD) instructions. For example, all Intel mobile CPUs with Xscale technique support integer MMX instructions that are originally designed for the Intel Pentium Pro desktop systems. These SIMD instructions can perform the same type of computation on multiple operands simultaneously. Although all operands are required to be integers, it is usually possible to use a sequence of integer SIMD instruction to perform real computation on several operands in a single instruction time. With the pseudo-floating-point SIMD technique, we successfully boosted the 3D graphics rendering operations of lighting and pixel interpolation. For specific mobile platforms, we can also make use of special features to accelerate graphics computation. Our 3D graphics development on the HP iPAQ PDA uses the Vincent library (http://ogl-es.sourceforge.net/), which is a software implementation of the OpenGL/ES standard, the OpenGL standard tailored for mobile devices. Because Vincent uses the Intel Graphics Performance Primitive (GPP) library, we can easily implement many graphics computing part of the program by the fixed-point SIMD technique. To implement 3D graphics engine on mobile phones is more challenging because of their CPUs even rarely support SIMD instructions. On the Nokia N70 (with Symbian operating system), we developed a voxel engine 4 , which is originally designed by game developers for low-end home computers in 1980’s. Different to the polygon-rasterizer engine as defined by the OpenGL/ES, the voxel engine is not capable of rendering full 3D objects represented by polygons, but it can render a height-map, which is a 2D function defined on the space of hx, yi and returning a height value z: z ← h(x, y) Because, generally, many 3D visual presentation are designed to layout most information on a 2D space to make the presentation easier to read by human users, other details are then added to the third dimension. For example, take a closer look at the 3D pathfinder networks (c.f. Figure 3), the spheres that represent users and the edges connecting these spheres are all defined on the hx, yi space. If we represent users by hemi-spheres instead of the spheres, as illustrated in Figure 8, we can simply model the visual presentation of pathfinder network as a height-map. Similarly, the cluster map can also be represented by a height-map.

6

Conclusions and Future Works

In this paper, we present what we learned from engineering several Web visualization systems on mobile devices as the MWeb3D framework. Our discussion covered four major aspects: (1) to address the restricted computational power 4

http://www.flipcode.com/articles/article mobilegfx01.shtml

Fig. 8. Model the pathfinder network by a height map.

of mobile device, we separate the visualization into two steps of generating visual presentations and rendering them. By moving distributing the separately tasks onto server systems, we organized an efficient Web visualization pipeline, that can significantly lighten the computation intensity of mobile devices. (2) To address the slow network connectivity, we design compact encoding schema to compactly encode the generated visual representations before transmitting them to the mobile devices for rendering and displaying. (3) To adapt to the small screen size and specific input methods of mobile devices, we discuss general interaction methods, especially the level-of-detail techniques, to modify existed visual representations. (4) We also discussed how to make full use of mobile facilities to implement highly efficient graphics rendering engine. Other than Web visualization, some of above summarizations are generally applicable to other mobile visualization applications. With the exciting development of mobile applications, we hope to further extend our exploration and experiences to more 3D visualization application on mobile devices.

7

Acknowledgment

This project is supported in part by the National Science Foundation of China No. 60573061 and Hong Kong RGC Project No. 1062/02E.

References 1. Cugini, J., Laskowski, S.: Design of 3D visualization of search results: Evolution and evaluation. In: Proc. of IST/SPIE’s 12th Annual International Symposium, San Jose, CA (2000) 23–28 2. Kules, B., Shneiderman, B.: Categorized graphical overviews for Web search results: An exploratory study using u.s. government agencies as a meaningful and stable structure. In: Proc. of the 3rd annual workshop on HCI Research in MIS. (2005) 3. Mann, T.M.: Visualization of www-search results. In: Proc. DEXA Workshop. (1999)

4. Roberts, J.C., Suvanaphen, E.: Visual bracketing for Web search result visualization. In et al, E.B., ed.: Proceedings Information Visualization (IV03), IEEE Computer Society (2003) 264–269 5. Wang, Y., Zhou, L.Z.: Extracting, presenting and browsing of Web social information. In: Proc. Web Age Information Management, LNCS 3739. (2005) 828–833 6. Chittaro, L.: Visualizing information on mobile devices. IEEE Computer 39(3) (2006) 40–45 7. Chen, C., Paul, R.J.: Visualizing a knowledge domain’s intellectual structure. IEEE Computer 34(3) (2001) 65–71 8. Shneiderman, B.: Tree visualization with tree-maps: 2D space-filling approach. ACM Transactions on Graphics 11(1)) (1992) 92–99 9. Bederson, B.B., Shneiderman, B., Wattenberg, M.: Ordered and quantum treemaps: making effective use of 2D space to display hierarchies. ACM Transactions on Graphics 21(4) (2002) 833–845 10. Fluit, C., Sabou, M., van Harmelen, F.: Cluter map — ontology-based information visualization. Chapter 3 of Visualizing the Semantic Web, editors Vladimir Geroimenko and Chaomei Chen (2002) 36–48 11. Bederson, B.B., Clamage, A., Czerwinski, M.P., Robertson, G.G.: Datelens: A fisheye calendar interface for pdas. ACM Transactions on Computer-Human Interaction 11(1) (2004) 90–119

2D/3D Web Visualization on Mobile Devices

Web visualization on both high-end and low-end mobile devices as the. MWeb3D ... lithically as a single piece of software running on a single computer.

2MB Sizes 7 Downloads 350 Views

Recommend Documents

Effective Web Searching on Mobile Devices
the time limit, and these were regarded as missing data. We chose the ... alternative visualization of page structure, such documents could be segmented at.

Interactive Mobile 3D Graphics for On-the-go Visualization and ...
Interactive Mobile 3D Graphics for On-the-go Visualization .... The M3G Model manages the data and behaviour of the 3D graphical ..... In Museums and the Web.

Diamond Browser: Faceted Search on Mobile Devices
image running on Amazon's EC2 environment and uses. Lucene/Solr, Lucidworks Enterprise, and SolrPHPClient to. HCIR 2011, October 20, 2011, Mountain ...

Digital Signature Verification on Mobile Devices - International Journal ...
qualified signature, mobile signature, SIM card, Java ME, signature services, mobile devices, .... Microsoft's cryptographic system basically consists of several components; ... out the following functions: the generation and exchange keys, data ...

Digital Signature Verification on Mobile Devices
With the purpose of having a solution that can be used in any mobile device, the e-signature solution should be independent of the operating system. • A user can have more than one identity. Therefore, it should be interesting that the solution cou

eyes-free text entry on mobile devices
GRADUATE PROGRAMME IN COMPUTER SCIENCE AND ENGINEERING ... This thesis explores eyes-free text entry solutions for mobile devices. For devices ...

Trustworthy Execution on Mobile Devices: What ...
assumed to be trustworthy, i.e., the software modules executing in the isolated envi- ... (SE) which provides isolated execution (§3), secure storage (§3), and a trusted path ..... solutions in the form of hypervisors or virtual machine monitors [3

pdf-0738\face-detection-and-recognition-on-mobile-devices-by ...
pdf-0738\face-detection-and-recognition-on-mobile-devices-by-haowei-liu.pdf. pdf-0738\face-detection-and-recognition-on-mobile-devices-by-haowei-liu.pdf.

Diamond Browser: Faceted Search on Mobile Devices
ABSTRACT. Faceted search interfaces are commonly used on the Web, especially on sites for online shopping, document collections, and library catalogs. As use of mobile web devices such as smartphones and tablet computers with smaller screens has incr

eyes-free text entry on mobile devices - Hussain Tinwala
I would also like to thank all my colleagues at the Interactive Systems ...... The term 'mobile phone' has been extensively used to refer to a range of different ...... competing products emerged, such as LG's Prada, Samsung's D988, Nokia's N95 ...

Programming Mobile Web - GitHub
Wordpress. Theme. Plugin. Joomla. Theme. Add on. Drupal. Module. Theme. More … Forum. Vanilla. esoTalk. Phpbb. More … More … Web server. Apache.

Mobile Devices and Student Innovators.pdf
... using mobile technology. Student work was evaluated to using SAMR.1. Work was published along the way on a professional. blog at KerryHawk02: Teaching ...

Speech Recognition for Mobile Devices at Google
phones running the Android operating system like the Nexus One and others becoming ... decision-tree tied 3-state HMMs with currently up to 10k states total.

Mobile devices and systems -
Windows 7, Mac OS 4 and Android. .... Latest version of Android is Android 2.2 (Froyo) which is based on Linux Kernel 2.6.32 and ... Adobe flash 10.1 support.

Mobile devices and systems -
Windows 7, Mac OS 4 and Android. 2.4 haNdheld deviCes. Not only has there been a transformation of titanic proportions in computing devices, but.

Motorola Mobile Devices calls up ... - Services
Mobile technology giant saves millions on legacy applications by using cloud-based tools to replace limiting email. Organization. =(0

Programming mobile devices - an introduction for practitioners.pdf ...
Programming mobile devices - an introduction for practitioners.pdf. Programming mobile devices - an introduction for practitioners.pdf. Open. Extract. Open with.

Web Page Switching on Mobile Browsers - Research at Google
May 2, 2013 - running Android 4.1 with a full-screen web-view .... “If I am on my phone, I want to see a lot of things quickly. [With] the ... CHI 2010, ACM Press.

Augmenting 3D urban environment using mobile devices
computer vision techniques to register the live view of the device with the 3D .... set of views that equally partition the 360 degrees angular range around the ...

Optimized Lightweight Thread Framework for Mobile Devices ...
are two software approaches to adjust the stack size: • Changing system ..... understanding the detailed interactions among threads to ana- lyze the purpose of ...