During the last few years mobile phones have been being replaced by new devices called smartphones. A more
intelligent'' version of a mobile phones, smartphones combine usual phoning’’ facilities with the functionality and performance of personal computers. Moreover, they are equipped with various sensors, such as camera and GPS, and are open to third-party applications. Being almost all the time with their users, it is not surprising that smartphones have access to very sensitive private data. Unfortunately, these data are of particular interest not only to the device owners. Developers of third-party applications embed data collection functionality either to feed advertising frameworks or for their own purposes. Moreover, there are also adversaries aiming at gathering personal user information or performing malicious actions. In this situation the users have a strong motivation to safeguard their devices from being misused and want to protect their privacy.Among all operating systems for mobile platforms, Android developed by Google is the recognized leader. This operating system is installed on four out of five new devices. In this thesis we propose a set of improvements to enhance security of the Android ecosystem and ensure trustworthiness of the applications installed on the device. In particular, we focus on the application ecosystem security, and research the following key aspects: identification of suspicious applications; application code analysis for malicious functionality; distribution of verified applications to end-user devices; and enforcing security on the device itself. It was previously shown that adversaries often relied on app repackaging to burst the proliferation of malicious applications. As the first contribution, this dissertation proposes a fast approach to detect repackaged Android applications. If a repackaged application is detected, it is necessary to understand whether it is malicious or not. Today Android malware conceal their malicious nature using dynamic code update techniques, thus, static analyzers cannot detect this vicious behavior. The second contribution of this work is a static-dynamic analysis approach to discover and analyse apps in the presence of dynamic code updates routines. To increase the user's confidence in the installed apps, as the third contribution we propose the concept of trusted stores for Android. Our approach ensures that a user can install only the applications vetted and attested by trusted stores. Finally, the forth contribution is the design and implementation of a policy-based framework for enforcing software isolation of applications and data that may help to improve the security of end-user devices.