Multimedia Mobile Application Development in iOS School of EECS - - PowerPoint PPT Presentation

▶
multimedia
SMART_READER_LITE
LIVE PREVIEW

Multimedia Mobile Application Development in iOS School of EECS - - PowerPoint PPT Presentation

Sep 22, 2022 132 likes •531 views

Multimedia Mobile Application Development in iOS School of EECS Washington State University Instructor: Larry Holder Mobile Application Development in iOS 1 Outline Audio recording, access, and playback Speech recognition and

slide-1
SLIDE 1

Multimedia

Mobile Application Development in iOS School of EECS Washington State University Instructor: Larry Holder

Mobile Application Development in iOS 1

slide-2
SLIDE 2

Outline

  • Audio recording, access, and playback

– Speech recognition and synthesis

  • Image capture, access, and display
  • Video recording, access, and playback

Mobile Application Development in iOS 2

slide-3
SLIDE 3

Audio Recording and Playback

  • Use AVFoundation framework
  • Configure AVAudioSession singleton class

– Need permission to access microphone

  • AVAudioSession.recordPermission == .granted
  • AVAudioSession.requestRecordPermission((Bool -> Void))

Mobile Application Development in iOS 3

import AVFoundation class ViewController: UIViewController {

  • verride func viewDidLoad() {

let session = AVAudioSession.sharedInstance() if session.recordPermission != .granted { session.requestRecordPermission({Bool in}) } } }

slide-4
SLIDE 4

Audio Recording and Playback

  • Configure AVAudioSession

– setCategory(category, mode, options) – Category (e.g., AVAudioSession.Category.playAndRecord) – Mode (e.g., AVAudioSession.Mode.spokenAudio) – Options (e.g., mixWithOthers, duckOthers, defaultToSpeaker)

  • setActive(Bool)

– Request access to audio hardware – May fail if higher-priority task using audio

Mobile Application Development in iOS 4

slide-5
SLIDE 5

Audio Recording and Playback

  • Configure and activate audio session

Mobile Application Development in iOS 5

let session = AVAudioSession.sharedInstance() do { try session.setCategory(AVAudioSession.Category.playAndRecord, mode: AVAudioSession.Mode.spokenAudio, options: [AVAudioSession.CategoryOptions.duckOthers, AVAudioSession.CategoryOptions.defaultToSpeaker]) try session.setActive(true) } catch { print("error starting audio session") }

slide-6
SLIDE 6

Audio Recording

  • Initialize

– AVAudioRecorder(url, settings) throws

  • Get URL to sound file in documents directory
  • Settings dictionary: Need at least AVFormatIDKey

– https://developer.apple.com/documentation/coreaudio/core_audio_data_types/157209 6-audio_data_format_identifiers

  • Main methods

– prepareToRecord(), record(), pause(), stop()

  • AVAudioRecorderDelegate methods

– audioRecorderDidFinishRecording

Mobile Application Development in iOS 6

Note: iOS simulator can access Mac's microphone.

slide-7
SLIDE 7

Audio Recording: Setup

Mobile Application Development in iOS 7

class ViewController: UIViewController, AVAudioRecorderDelegate { let audioFile = "audioFile.m4a" var audioFileURL: URL! var audioRecorder: AVAudioRecorder?

  • verride func viewDidLoad() {

// Get URL to audio file let paths = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask) let docDir = paths[0] audioFileURL = docDir.appendingPathComponent(audioFile) // Setup audio recorder let settings = [AVFormatIDKey: kAudioFormatMPEG4AAC] do { audioRecorder = try AVAudioRecorder(url: audioFileURL, settings: settings) audioRecorder?.delegate = self } catch { print("error creating audio recorder") } } }

slide-8
SLIDE 8

Audio Recording

  • AVAudioRecorderDelegate method

Mobile Application Development in iOS 8

func startRecording() { audioRecorder?.record() } func stopRecording() { audioRecorder?.stop() } func audioRecorderDidFinishRecording(_ recorder: AVAudioRecorder, successfully flag: Bool) { if flag { print("recording successful") } else { print("recording failed") } // Modify view: Change "Stop" to "Start" }

slide-9
SLIDE 9

Audio Playback

  • Initialize

– AVAudioPlayer(url) throws

  • Get URL to sound file in documents directory
  • Must be reinitialized when sound file rewritten
  • Main methods

– prepareToPlay(), play(), pause(), stop() – currentTime – set to 0 to return to front

  • AVAudioPlayerDelegate methods

– audioPlayerDidFinishPlaying

Mobile Application Development in iOS 9

slide-10
SLIDE 10

Audio Playback: Setup

Mobile Application Development in iOS 10

class ViewController: UIViewController, AVAudioPlayerDelegate { var audioFileURL: URL! var audioPlayer: AVAudioPlayer? func startPlaying() { // Setup audio player do { audioPlayer = try AVAudioPlayer(contentsOf: audioFileURL) audioPlayer?.delegate = self } catch { print("error accessing audio player") } audioPlayer?.play() } func stopPlaying() { audioPlayer?.stop() } }

slide-11
SLIDE 11

Audio Playback

  • AVAudioPlayerDelegate method

Mobile Application Development in iOS 11

func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) { if flag { print("playback finished") } else { print("playback error") } // Modify view: Change "Stop" to "Start" }

slide-12
SLIDE 12

Accessing Audio Library

  • Maintain privacy and DRM
  • MediaPlayer framework

– MPMediaPickerController to select audio – MPMediaPickerDelegate

  • mediaPicker(didPickMediaItems)
  • mediaPickerDidCancel

– MPMediaPlayerController.applicationQueuePlayer

  • setQueue(mediaItemCollection)
  • play(), pause(), stop()

Mobile Application Development in iOS 12

Note: No music library in iOS simulator. Need a real device.

slide-13
SLIDE 13

Accessing Audio Library

Mobile Application Development in iOS 13

import MediaPlayer class ViewController: UIViewController, MPMediaPickerControllerDelegate { var mediaPlayer = MPMusicPlayerController.applicationQueuePlayer @IBAction func selectSongToPlayTapped(_ sender: UIButton) { let mediaPicker = MPMediaPickerController(mediaTypes: .anyAudio) mediaPicker.delegate = self present(mediaPicker, animated: true, completion: {}) } func mediaPicker(_ mediaPicker: MPMediaPickerController, didPickMediaItems mediaItemCollection: MPMediaItemCollection) { mediaPlayer.setQueue(with: mediaItemCollection) mediaPicker.dismiss(animated: true, completion: {}) } func mediaPickerDidCancel(_ mediaPicker: MPMediaPickerController) { mediaPicker.dismiss(animated: true, completion: {}) } }

slide-14
SLIDE 14

Speech Recognition

  • Speech framework
  • SFSpeechRecognizer

– requestAuthorization()

  • SFSpeechRecognizerDelegate

– speechRecognizer(availabilityDidChange)

Mobile Application Development in iOS 14

slide-15
SLIDE 15

Speech Recognition: Setup

Mobile Application Development in iOS 15

import Speech class ViewController: UIViewController, SFSpeechRecognizerDelegate { var speechRecognitionAllowed = false var speechRecognizer: SFSpeechRecognizer?

  • verride func viewDidLoad() {

super.viewDidLoad() SFSpeechRecognizer.requestAuthorization(handleAuth) } func handleAuth (status: SFSpeechRecognizerAuthorizationStatus) { switch status { case .authorized: speechRecognitionAllowed = true speechRecognizer = SFSpeechRecognizer() speechRecognizer?.delegate = self default: speechRecognitionAllowed = false } }

slide-16
SLIDE 16

Speech Recognition: Delegate

Mobile Application Development in iOS 16

// Delegate method func speechRecognizer(_ speechRecognizer: SFSpeechRecognizer, availabilityDidChange available: Bool) { if available { speechRecognitionAllowed = true } else { speechRecognitionAllowed = false } }

slide-17
SLIDE 17

Speech Recognition

Mobile Application Development in iOS 17

var spokenText: String = "Hello, world." func recognizeSpeech () { if speechRecognitionAllowed { let request = SFSpeechURLRecognitionRequest(url: audioFileURL) request.shouldReportPartialResults = false speechRecognizer?.recognitionTask(with: request, resultHandler: speechRecognitionHandler) } } func speechRecognitionHandler (result: SFSpeechRecognitionResult?, error: Error?) { if let res = result { self.spokenText = res.bestTranscription.formattedString print("spoke: \(self.spokenText)") } else { print("speech recognition error") } }

slide-18
SLIDE 18

Speech Synthesis

Mobile Application Development in iOS 18

import AVFoundation func synthesizeSpeech() { let speechSynthesizer = AVSpeechSynthesizer() let utterance = AVSpeechUtterance(string: self.spokenText) // utterance.voice = AVSpeechSynthesisVoice(identifier: "...") speechSynthesizer.speak(utterance) }

slide-19
SLIDE 19

Images and Video

  • Add privacy properties for access to camera and photo library

– Request authorization

  • UIImagePickerController

– Take a picture or video – Select a picture or video from library

  • AVFoundation framework

– Lower-level control of image and video assets

Mobile Application Development in iOS 19

Note: iOS simulator cannot access Mac

  • camera. Need real

device for testing. Can drag-and-drop images and videos into Photos app on iOS simulator.

slide-20
SLIDE 20

UIImagePickerController: Properties

  • allowsEditing
  • sourceType: .photoLibrary, .savedPhotosAlbum, .camera
  • mediaTypes

– kUTTypeImage as String, kUTTypeMovie as String

  • import MobileCoreServices

– UIImagePickerController.availableMediaTypes(for)

Mobile Application Development in iOS 20

slide-21
SLIDE 21

UIImagePickerController: Delegates

  • UIImagePickerControllerDelegate

– imagePickerController(didFinishPickingMediaWithInfo info)

  • info[UIImagePickerController.originalImage]
  • info[UIImagePickerController.editedImage]

– imagePickerControllerDidCancel

  • UINavigationControllerDelegate

– Required, but used implicitly

Mobile Application Development in iOS 21

slide-22
SLIDE 22

Select Image from Photo Library

Mobile Application Development in iOS 22

import UIKit import MobileCoreServices // to get kUTTypeImage class ViewController: UIViewController, UIImagePickerControllerDelegate, UINavigationControllerDelegate { var selectedImage: UIImage? func selectImageFromLibrary () { if UIImagePickerController.isSourceTypeAvailable(.photoLibrary) { let picker = UIImagePickerController() picker.delegate = self picker.allowsEditing = false picker.sourceType = .photoLibrary picker.mediaTypes = [kUTTypeImage as String] self.present(picker, animated: true, completion: nil) } else { print("photo library not available") } } }

slide-23
SLIDE 23

Select Image from Photo Library

Mobile Application Development in iOS 23

// UIImagePickerControllerDelegate methods func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey: Any]) { selectedImage = info[.originalImage] as? UIImage imageView.image = selectedImage picker.dismiss(animated: true, completion: nil) } func imagePickerControllerDidCancel(_ picker: UIImagePickerController) { picker.dismiss(animated: true, completion: nil) }

slide-24
SLIDE 24

Take Picture

Mobile Application Development in iOS 24

func takePicture () { // same as selectImage, except use sourceType .camera if UIImagePickerController.isSourceTypeAvailable(.camera) { let picker = UIImagePickerController() picker.delegate = self picker.allowsEditing = false picker.sourceType = .camera picker.mediaTypes = [kUTTypeImage as String] self.present(picker, animated: true, completion: nil) } else { print("camera not available") } }

slide-25
SLIDE 25

Save Image to File

Mobile Application Development in iOS 25

func writeImage(_ image: UIImage, to fileName: String) { if let directoryURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first { let fileURL = directoryURL.appendingPathComponent(fileName) // Can also use image.pngData, but rotates image if let imageData = image.jpegData(compressionQuality: 1.0) { do { try imageData.write(to: fileURL) } catch { print("\(error)") } } else { print("Unable to convert image to jpeg.") } } else { print("Error accessing document directory.") } }

slide-26
SLIDE 26

Read Image from File

Mobile Application Development in iOS 26

func readImage(from fileName: String) -> UIImage? { if let directoryURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first { let fileURL = directoryURL.appendingPathComponent(fileName) do { let imageData = try Data(contentsOf: fileURL) let image = UIImage(data: imageData) return image } catch { print("\(error)") } } else { print("Error accessing document directory.") } return nil }

slide-27
SLIDE 27

Save Image to Photo Library

  • UIImageWriteToSavedPhotosAlbum(image: UIImage,

completionTarget: Any?, // self completionSelector: Selector?, // #selector(imageWriteHandler) contextInfo: UnsafeRawPointer?) // nil

  • func imageWriteHandler(image: UIImage,

didFinishSavingWithError error: Error?, contextInfo: UnsafeRawPointer?)

Mobile Application Development in iOS 27

slide-28
SLIDE 28

Save Image to Photo Library

Mobile Application Development in iOS 28

func saveImage () { if let image = selectedImage { UIImageWriteToSavedPhotosAlbum(image, self, #selector(imageWriteHandler), nil) } } @objc func imageWriteHandler(_ image: UIImage, didFinishSavingWithError error: Error?, contextInfo: UnsafeRawPointer?) { if let err = error { print("error saving image: \(err.localizedDescription)") } else { print("image saved to camera roll") } }

slide-29
SLIDE 29

Working with Video

  • Selecting video from library similar to images

– Still use UIImagePickerController – Use mediaType kUTTypeMovie

  • Delegate method gets URL to video
  • Play video using AVPlayer and AVPlayerViewController
  • Recording video similar to images

– Authorize use of microphone

  • Saving video similar to images

Mobile Application Development in iOS 29

slide-30
SLIDE 30

Select Video from Library

Mobile Application Development in iOS 30

import UIKit import MobileCoreServices // to get kUTTypeMovie class ViewController: UIViewController, UIImagePickerControllerDelegate, UINavigationControllerDelegate { var videoURL: URL? func selectVideo () { if UIImagePickerController.isSourceTypeAvailable(.photoLibrary) { let picker = UIImagePickerController() picker.delegate = self picker.allowsEditing = false picker.sourceType = .photoLibrary picker.mediaTypes = [kUTTypeMovie as String] self.present(picker, animated: true, completion: nil) } else { print("video library not available") } }

slide-31
SLIDE 31

Select Video from Library

Mobile Application Development in iOS 31

// UIImagePickerControllerDelegate methods func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey: Any]) { videoURL = info[.mediaURL] as? URL picker.dismiss(animated: true, completion: nil) } func imagePickerControllerDidCancel(_ picker: UIImagePickerController) { picker.dismiss(animated: true, completion: nil) }

slide-32
SLIDE 32

Play Video

Mobile Application Development in iOS 32

import AVKit // for AVPlayerViewController import AVFoundation // for AVPlayer func playVideoInNewView () { if let url = videoURL { let playerView = AVPlayerViewController() playerView.player = AVPlayer(url: url) present(playerView, animated: true, completion: nil) } } @IBOutlet weak var videoView: UIView! func playVideoInSubView () { if let url = videoURL { let playerView = AVPlayerViewController() playerView.player = AVPlayer(url: url) playerView.view.frame = videoView.frame self.addChild(playerView) // initialize player view self.view.addSubview(playerView.view) // display player view } }

slide-33
SLIDE 33

Record Video

Mobile Application Development in iOS 33

func recordVideo () { // same as selectVideo, except use sourceType .camera if UIImagePickerController.isSourceTypeAvailable(.camera) { let picker = UIImagePickerController() picker.delegate = self picker.allowsEditing = false picker.sourceType = .camera picker.mediaTypes = [kUTTypeMovie as String] self.present(picker, animated: true, completion: nil) } else { print("camera not available") } }

slide-34
SLIDE 34

Save Video to File

Mobile Application Development in iOS 34

let videoFileName = "myVideo.mov" // .mov extension important func writeVideo(_ videoURL: URL?, to fileName: String) { if let directoryURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first { let fileURL = directoryURL.appendingPathComponent(fileName) if let vidURL = videoURL { do { let videoData = try Data(contentsOf: vidURL) try videoData.write(to: fileURL) } catch { print("\(error)") } } } else { print("Error accessing document directory.") } }

slide-35
SLIDE 35

Read Video from File

Mobile Application Development in iOS 35

// Just return URL to video file func getVideoFileURL(from fileName: String) -> URL? { if let directoryURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first { let fileURL = directoryURL.appendingPathComponent(fileName) return fileURL } else { print("Error accessing document directory.") } return nil }

slide-36
SLIDE 36

Save Video to Library

  • UISaveVideoAtPathToSavedPhotosAlbum(videoPath: String,

completionTarget: Any?, // self completionSelector: Selector?, // #selector(videoWriteHandler) contextInfo: UnsafeRawPointer?) // nil

  • func videoWriteHandler(videoPath: String,

didFinishSavingWithError error: Error?, contextInfo: UnsafeRawPointer?)

Mobile Application Development in iOS 36

slide-37
SLIDE 37

Save Video to Library

Mobile Application Development in iOS 37

func saveVideoToLibrary () { if let videoPath = videoURL?.path { UISaveVideoAtPathToSavedPhotosAlbum(videoPath, self, #selector(videoWriteHandler), nil) } } @objc func videoWriteHandler(_ videoPath: String, didFinishSavingWithError error: Error?, contextInfo: UnsafeRawPointer?) { if let err = error { print("error saving video: \(err.localizedDescription)") } else { print("video saved to library") } }

slide-38
SLIDE 38

Resources

  • AVFoundation and AVKit (more media tools here)

– developer.apple.com/avfoundation/ – developer.apple.com/documentation/avkit

  • MediaPlayer framework

– developer.apple.com/documentation/mediaplayer

  • Speech framework

– developer.apple.com/documentation/speech

  • UIImagePickerController

– developer.apple.com/documentation/uikit/uiimagepickercontroller

Mobile Application Development in iOS 38