PDF draft (for internal review use only): exp_audio-haptic_tap.pdf


PDF version: d4.2.pdf (~4Mb)


PDF version: bresindewittpapetticivolanifontana_ison2010.pdf (~6Mb)


PDF version: paper.pdf


PDF version: del5.1.pdf (~1.7Mb)


PDF proof: proof25.pdf (~2Mb)


Latest: proof.pdf (~1Mb)


--- MINUTES FROM THE *NIW* MEETING ---

9-10 JUNE 2010

McGill University
Center for Intelligent Machines (CIM)
McConnell Engineering Building
3480 University St. - 4th floor

Author: F. Fontana

JUN. 9 - The meeting starts at 8.45am, with coffee break and work to start video conferencing with AAU and UPMC. (Previous work made to test configurations for the connections between European delegates participating from remotely and CIM started on June 7; overall, it took a couple of hours)

Participants:

McGill: Y. Visell, J. Cooperstock, G. Millet, G. Chaw, M. Otis, S. Smith.
UNIVR: F. Fontana, S. Papetti, M. Civolani
INRIA: G. Cirio, M. Marchal
UPMC: A. Berrezag, V. Hayward (both in teleconference)
AAU: S. Serafin (in teleconference)

9.15am - Tour the table. Professor Paul Kry from McGill is welcome as an external attendant.

9.30am - McGill Research update

- Ground perception and vibration: Synthesis of microscopic texture through the tiles.
- New experiments: Perception of stiffness. Tangentially force-actuated vibrotactile tiles may be researched in future months.
- Compensated tiles: Force equalization of the tile physical structure and user's load on the tile.
- Interaction with cartographic data: A demo will be shown in the afternoon.
- Target selection with a visual floor touch surface.
Several questions are addressed to the McGill presentations.

10.30am - INRIA Research and activity update

- Recent student exchanges are listed.
- Results from the NIW tutorial at the IEEE VR2010 conference are briefly summarized.
- Visual simulation of slopes: Varying the first-person height gives promising results. Conversely, varying only the horizontal velocity with steepness was not successful. The joint provision of the two variation is effective.
- Early simulation of liquid interactions (with McGill): Walking over liquids using physically-based sounds is the research task of GC during his staying at McGill. Results are shown already proving the potential of the method, although in non-NIW interaction contexts.
- Separation of Time-Frequency patterns in sounds (with UNIVR): The experiment is robust, but the saliency of the results does not clearly point to a neat conclusion in terms of subjective tendencies.
- Ground affordances: Presentation of the final results from the experimental analysis.
Several questions are addressed to the INRIA presentations.

11.30am - Short break

11.45am - UNIVR Research update

- Current state of the active shoes: Improvements in the acquisition and reliability of sampling of the input data through the generation of new software for the Arduino card.
- Audio-haptic experiment and illusion: Presentation of the hypothesis, methodology, and protocol. Early results from the experimental analysis are very promising.

12.15pm Lunch

2.30pm Demos of the facilities at CIM are shown to the attendants.

3.15pm Project status

- Per the contacts with the participants, and after the morning review of the current research, FF thinks that the project is healthy in terms of technological and research advancements. A deeper analysis will be attempted later on this meeting, after having seen also the presentations from AAU and UPMC.

- FF reads the summary of the final first review report of the NIW project, consolidated in Paris. Among all the raised issues, one seems to have been still left partially unanswered by the consortium, that is, the reviewers' suggestion to perform an experiment comparing the shoe against the tiles. The recent experiments made at McGill and UNIVR dealing with the illusory cues conveyed by the two designs represent a solid point, though, suggestions are solicited to be produced later on this meeting.

3.30pm Plans for the future (review) meeting

-The following dates are sketched:
Oct. 11-13 (best choice for JC), Nov. 1-5, Nov. 8-10, Nov. 15-16
MM will launch a doodle poll as soon as possible

4.00pm Dates of delivery/reporting and content of deliverables/periodic report

- FF reminds that this year a more rigorous scheduling of deliverable dates should be respected. He proposes the following dates:
mid September: advanced drafting of all deliverables
end of September: financial documentation ready (form C's + person/month of all participants)
first half of October: advanced drafting of the periodic report

- YV and FF pass through the titles and content of the deliverables for the second period

- YV calls for a minor re-scheduling of next day, due to temporal constraints of the remote attendants from Europe. In particular, the next day includes possibility to join a seminar at McGill from 12am to 1pm, on augmented reality in simulation of space explorations. All attendants show interest for this opportunity.

The meeting closes at 5.00pm.

Dinner scheduled at 7pm at "Le P'Tit Plateau"


JUN. 10 - The meeting starts at 9.30am

Participants:

McGill: Y. Visell, J. Cooperstock, G. Millet
UNIVR: F. Fontana, S. Papetti, M. Civolani
INRIA: G. Cirio, M. Marchal
UPMC: V. Hayward (in teleconference)
AAU: S. Serafin (in teleconference)

9.35am - Discussion on the book deliverable

- YV summarizes the aims and contents of the open book. Conclusions are finally drawn concerning key aspects of the edition:
Editors: FF, YV
List of contents (drawn from the Annex I of the project)
* Introduction: FF and YV lead
* Hardware: VH lead
* Sensing: YV lead
* Feedback: FF and SS lead
* Pseudo-haptics and interaction techniques: AL and MM lead
* Integration and presence: Rolf Nordahl lead
Publisher: Logos-Verlag, or Open publishing line at Springer? SS will investigate possibilities for the latter option
Chapters to be exported to a future Springer book: YV will coordinate this activity, if any

10.15am - Research exchanges

- Recent and current exchanges are listed:
SS reports of: AAU -> INRIA (Luca Turchet); AAU -> UPMC (SS and Kristina Daniliauskaite)
YV reports of: UPMC -> McGill (Nizar Ouarti); INRIA -> McGill (GC); UNIVR -> McGill (Stefano Zambon)

- Possible future exchanges between UNIVR and INRIA are proposed by MM

10.35am Management

- FF updates on current status of the project management. As the consortium already knows, the first payment has arrived beginning of May. The first iteration with the Commission for changing the coordinator is in due course. FF hopes to conclude this process as soon as possible. All participants have nothing to question, and leave the current coordinator to act on behalf of the consortium.

10.50am - AAU Research update

- SS, from remotely, reports on the migration of AAU shoe models from acoustic to force sensing. Three experimental studies based on comparison of user's performances with different feedback modalities during a task are being performed, and their results will be provided when ready.

- SS leaves the remote connection at 11:05

11.10am - UPMC Research update

- VH reports on newly engineered force actuators containing smaller magnets/components that are ready for delivery, as well as on improvements to the membrane-based actuators, that are yet not ready to be used.

- Since it is of chief interest for the Consortium to have more actuators at hand, FF asks VH to prepare an offer for a small cost-effective mass production of the new force actuators (100 pieces?), to be distributed among beneficiaries by providing from the consortium part the necessary economical support to UPMC to cover costs of the material.

- VH says there may be collapsing with existing patents. He will take care of checking out possible issues in this sense.

- VH reports that there are fair results with the current experiment. The related experimental hypothesis may thus be abandoned.

- SP and FF solicit the creation of kind of installation guidelines, useful to optimally install the actuators in the shoe soles in a way to obtain an as most stable behavior.

The attendants move to the seminar at 11.50am.

1.15pm - Lunch

The meeting starts again at 2pm. Teleconferences are over. The remaining attendants brainstorm on future research.

- FF presents the research activity and interests of Stefano Zambon, who will be visiting McGill from July 2010 to December 2010. The following points are discussed with YV and JC:
* Modeling and simulating distributed shoe/material contacts as point-wise interactions. Ex.: granular materials against the sole.
* Implementing such models in multi-core and/or GPU environment-based computing facilities at McGill.
* Collaborating with GC in the modeling and implementation of distributed interactions between a sole and the aforementioned particle-based fluid model.

- FF solicits ideas for realizing a comparison between tiles and shoes, as prescribed by the project reviewers. Best would seem to match the tactile displacements in the two interfaces, and then compare sensations of simple Poisson-generated granular textures. Such an experiment has a clear counterpart with equivalent tests on perceived roughness by the hands. The problem is now where to perform this comparison: either at McGill or INRIA, or at UNIVR, provided that a second shoe prototype could be delivered to such two places or conversely a pair of tiles of the latest generation could be built at UNIVR. Ways to solve this logistic issue will be discussed shortly after the meeting.


The meeting closes at 16.45


Download from here: niw_1_7.bib.zip


--- MINUTES FROM THE *NIW* MEETING ---

17-18 FEBRUARY 2010

University of Verona
Dipartimento di Informatica
Strada Le Grazie 15 - Verona 37134
Sala Riunioni, 2nd floor

Author: F. Fontana


FEB. 17 - The meeting starts at 3.00pm, with coffee break and work to start video conferencing with McGill University and INRIA

Participants:

UNIVR: F. Morreale (in teleconference from INRIA), F. Fontana, M. Civolani, S. Papetti, D. Rocchesso
McGill University: Y. Visell (in teleconference), G. Millet
AAU: R. Nordahl, S. Serafin, K. Daniliauskaite.
INRIA: A. Lecuyer, M. Marchal, L. Terziman, T. Le Bouffant (all in teleconference), T. Regia Corte
UPMC: V. Hayward

TokBox is used to connect with video. Skype is used for speech. Together they create a good video-conference link. At 3.40pm the Coordinator opens the meeting. Tour the table.

4.00pm - Management issues

- FF communicates the need to change the project coordinator. He explains that he has been very recently affiliated to the Dipartimento di Matematica e Informatica (DIMI) at the University of Udine, Italy. Hence, an amendment to the grant agreement will be proposed as soon as possible to the Commission, aiming to include DIMI as sixth beneficiary in the project as well as new coordinator, with minor research tasks. The core Italian research will remain at UNIVR. DIMI will draw funds exclusively from UNIVR. All the other beneficiaries' budgets will be left untouched from this operation.

- FF communicates that the coordinator change will imply a delay of the payment, otherwise in due course. This delay at the moment is unpredictable, and may amount to some months. Unfortunately, the payment process could not be started immediately after the end of the reporting period for issues related to the financial reporting of McGill University, that were solved only at the second half of January. (FF shortly explains what happened in December and January with McGill University). Even more, the coordination change cannot be postponed to the payment process since UNIVR has changed bank account in the meanwhile for internal reasons, this fact in its turn calling for an amendment to the grant agreement. The consortium, in conclusion, is made aware of the reasons that will cause the delay in the payment.

- FF makes circulate the official letter, through which Verona will start the amendment process. Notwithstanding any objection on its content, the letter will be sent to the officer on February 22nd, 2010. He then asks all beneficiaries to take the right to proceed with the amendment process also on their behalf. All beneficiaries agree on giving the coordinator the right to proceed with the change of coordinator on their behalf.

- AL will inquire whether the beneficiaries have to start any action related with the change of coordinator. FF believes they do not, meanwhile he gives mandate to AL to make this investigation on behalf of all beneficiaries, and possibly provide feedback on this.

- FF, in an evening email addressed to the consortium, adds one more point concerning the efficiency of project administration: "Also per the claim coming from the officer's and reviewers' official feedback, next year we will arrive to the review meeting with both the periodic report and the form C virtually completed, especially if the meeting takes place not too early as it perhaps happened last year.

This effort requires two basic facts:

i) shared activity involving everybody while preparing the periodic report (say, this year I have paved the way and fortunately the report has been judged positively, that is, we have a usable priority)

ii) less mistakes from some administrations while finalizing the financial forms (we know that some documents required many iterations with the coordinator, for trivial but substantial reasons like wrong sums, asynchronisms with the tables in the periodic report, mistaken signatures and so on). This is even more important, as once in Udine I will hardly find an error-free machine like Caterina when checking the correctness of the documentation. Please involve with this your administrations accordingly."

- FF opens the meeting of the General Assembly. He claims that the Consortium Agreement must be modified upon approval of the new coordinator. However, changes will be minor, holding the stability of the research workplan. Then, the two-month delay for public disclosure of the deliverables is analyzed, as it appears from the project evaluation document fed to the consortium by the project officer and reviewers after the end of the first period. VH proposes to ask, in occasion of the next delivery time, for an extension of the currently granted delay to four months, to better accommodate the International patenting rules and temporal processes. The assembly agrees. The meeting of the General Assembly is closed.

- FF asks for making plans for the next meeting. YV confirms the availability of McGill University to host it. The first half of June 2010 seems to best fit with the various beneficiaries' commitments. A Doodle poll is opened, to set a precise date.

5.00pm - Dissemination issues

- FF re-opens the discussion on the book deliverable. YV, who has played a prominent role in drafting content and editorial possibilities, is in favor of a linear application of the book roadmap that has been already proposed in the project technical annex (open content, publishing on demand, NIW-related topics and products). This said, editors should candidate from within the consortium. In parallel to this, AL reminds that contacts with researchers who were involved in Cyberwalk will be possible, during a forthcoming special session in which he participates at the next IEEE VR conference, to be held in Boston in March 20. Such contacts may fertilize a broader activity for the book. RN emphasizes that an enlargement of the book activity makes sense especially if the new researchers can take substantial part of the needed effort into charge. The consortium decides to give AL mandate to i) talk to the external researchers in Boston, to understand if the are really interested, and ii) if so, to propose the NIW book roadmap and a preliminary table of contents as a starting point for any discussion follow-up. This table of contents will be prepared by FF and YV in due time. Should such a discussion grow up, then FF will contact the officer to ask if the disclosure level of the book deliverable, now public, could be arranged in a way to account for the rights of publishers potentially interested in distributing the book.

In a later email addressed to the consortium, YV further points out: "My own suggestion would be to follow a path of least resistance, and produce the required NIW deliverable book in a narrowly-scoped, self-published way to circumvent problems from the EC (scheduling, approvals from project officer, etc), along the lines already encountered. It might include content closely related to the topics of the WPs in NIW, and provide a nice record of the project contributions and background to them, similar to the Sounding Object book.

I suggest we separately pursue a more broadly-scoped book project aimed for publication after 2011 -- i.e. on a looser schedule outside of the EC / FET / NIW, with involvement of external experts. This would provide time to solicit strong outside contributions, find a good publisher, and flexibility to deal with changes in timeline, contents, etc. If the two books are distinguished in scope, they can avoid competing for inputs. I've talked to some researchers that may be interested in contributing... "

- AL invites the consortium to collect slides and other presentation material, that is worth being presented at IEEE VR during the aforementioned special session.

- FF reminds that mobility of researchers must be annotated in the web site, by just adding a row in the related text file. At the moment there is a lot of such activity: FM@UNIVR is at INRIA, TRC@INRIA is at UNIVR, LT@AAU will soon go INRIA, Gabriel Cirio@INRIA will soon go to McGill University, Stefano Zambon@UNIVR will soon go to McGill University. Furthermore, exchange of prototypes has successfully been done: McGill University's tiles are at UNIVR and being further researched at UPMC, UPMC actuators have been successfully mounted in shoes at AAU and UNIVR.

5.40pm - Preliminary open discussion

- FF calls for some preliminary open discussion to be attempted, so as to take maximum advantage of the remote attendees in video conference. AL reminds that during the second period the project must start to provide substantial feedback on: common vision, common activities, common evaluation. He summarizes that evaluation may be presented at the next review meeting across topics, and not across beneficiaries. FF says that the three points envision an already promising track, especially if read the other way round: common evaluations are already taking place in the consortium; common activity should hopefully spring out from current research visits; still, there is partial lack of a common vision, even if it may finally arise as inductive consequence of the outcomes from common evaluations and activities. AL recalls three broad topics that had been envisioned in Paris: medicine/rehabilitation, architecture, gaming. FF reminds he had sketched the idea of working around a "virtual pathway" concept. YV proposes to frame the research made by NIW under the umbrella of Augmentation of kinesthetic information in interactions, where the interface does not display physical movement and locomotion, like in treadmills and other robotic devices.

6.15pm - The meeting is closed. Attendees meet at 8.00pm at the Trattoria Pero d'Oro for dinner. P. Cesari (UNIVR) and L. Turchet (AAU) join for the dinner as well.


FEB. 18 - The meeting starts at 10.30am

Participants:

UNIVR: F. Fontana, M. Civolani, S. Papetti, P. Cesari, P. Polotti
McGill University: G. Millet
AAU: R. Nordahl, S. Serafin, K. Daniliauskaite. L. Turchet
INRIA: T. Regia Corte
UPMC: V. Hayward

10.45pm - Scientific presentations

- UNIVR presents i) advancements made on the control of data flows in the Arduino board, necessary to record consistent pressure signals from the shoe prototype; ii) the state of advancement of the sonic floor, now detecting (yet offline) an approaching walker with reasonable accuracy; iii) the tile-based experiment on virtual skateboarding, ready to be performed at PC's lab once the experimental protocol is ready

- UPMC presents actuators using new, more powerful motors capable of exerting forces amounting to 7g. Ongoing works are aiming at: i) making such devices more compact, in a way to become able to put several actuators underfoot, probably in direct contact with feet rather than beneath the sole; ii) designing distributed soles, made of dense spatial 2D arrays of so-called "1-bit" (i.e. on/off) vibrotactile actuators. Attention to power consumption is paid as well, and zero-power actuators are under development which may become a reality in the near future. Finally, subjective experiments on the physiological effects of phase-delaying impulsive signals that are injected into the tiles are being performed, in terms of measured dynamic stance/equilibrium parameters of subjects standing on the tiles.

- McGill University introduces several activities going on, ranging from improvements to the active tiles to data browsing applications of the foot-based interface, running in the multimodal VR environment at the Shared Lab.

- AAU shows experiments on auditory perception of basic floor properties using the sound models developed for the project, with related quantitative results. However encouraging, such results in particular demonstrate that the addition of auditory environmental context helps subjects discriminate among properties even more accurately. Finally, KD introduces herself and her activities in AAU, in particular those overlapping with the project.

- INRIA presents current experimental joint activity with UNIVR in the discrimination of floor properties using reconstructions of footstep sounds made with different temporal and spectral mixtures, including hybrid mix-ups.

1.00pm - Lunch at Filu' Restaurant

2.30pm - Visit to the UNIVR lab

- The planned one-hour closing video-conference with McGill University and INRIA cannot take place, due to unexpected connection problems.

- The tile-based experiment on skateboarding is demoed. Almost all attendees try the demo, based on a pre-synthesized sound simulating 4 rolling wheels in continuously varying configurations, like accelerating/coasting/jumping/stopping. As a test, subjects are fed with a loud brown noise signal through headphones in order to cancel the audio feedback while retaining the vibrotactile cues. RN suggests to feed participants with something more surprising / hardly given for granted, as a sudden stop just after a fast coasting phase. This should give rise to less predictable behaviors of the subjects.

- The audio-haptic shoes are shown. They are received quite successfully, considering the wide range of foot sizes. Most participants enjoyed the simulations (creaking floor, snow, mud). In some cases the latency between the gesture and the feedback appeared to be too high, but that was due to incorrect calibration of the sensors for those particular feet. Some preliminary pseudo-haptic impressions are demonstrated, along with switch-offs of the low frequencies, resulting in the system to appear less realistic and effective.


The meeting closes at 4.30pm